Data synchronization method, device and equipment, readable storage medium and program product

Abstract

The invention relates to a data synchronization method and device, equipment, a readable storage medium and a program product, the data synchronization method and device are applied to a master cluster in a distributed storage system, the distributed storage system comprises the master cluster and a slave cluster, and the method comprises the steps that firstly, target data needing to be processed of the distributed system are obtained, and then the target data are sent to the slave cluster; storing the target data into a first memory space of the master cluster, and finally, synchronously writing the target data stored in the first memory space into the same position of a second memory space of the slave cluster based on a remote direct memory access technology; wherein the first memory space and the second memory space are mirror image memory spaces. By adopting the method, the data transmission delay and the CPU overhead can be reduced, and the data synchronization efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data synchronization method, apparatus, device, readable storage medium, and program product. Background Art

[0002] During business operation, due to business needs, business data needs to be written to multiple clusters. For example, business data is written to both ends of the master and slave clusters. When a problem occurs in one of the clusters, the user business can use the other cluster to continue providing services, thereby ensuring the continuity of the user business.

[0003] Currently, in conventional master-slave cluster data synchronization solutions, the business end copies data to the network layer of the master cluster. The master cluster then synchronizes the data to the slave cluster via the TCP (Transmission Control Protocol) network. The slave cluster then copies the data from the network, performs business processing based on the data, and then returns the processed results to the master cluster through a control flow, completing the business processing. This method of master-slave cluster data synchronization is inefficient. Summary of the Invention

[0004] Based on this, it is necessary to provide a data synchronization method, device, equipment, readable storage medium and program product that can improve the efficiency of master-slave cluster data synchronization in response to the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method, which is applied to a master cluster in a distributed storage system, where the distributed storage system includes a master cluster and a slave cluster. The method includes:

[0006] Obtain target data, which is the data that the distributed storage system needs to process;

[0007] Storing the target data in the first memory space of the main cluster;

[0008] Based on remote direct memory access technology, target data stored in the first memory space is synchronously written to the same position in the second memory space of the slave cluster; wherein the first memory space and the second memory space are mirror memory spaces.

[0009] In the above embodiment, a master cluster is applied to a distributed storage system, and the distributed storage system includes a master cluster and a slave cluster. The method includes: first, obtaining target data that the distributed system needs to process, then storing the target data in the first memory space of the master cluster, and finally, synchronously writing the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces. In this way, when the master cluster and the slave cluster synchronize data, the master cluster saves the target data in the first memory space, and then writes the target data in the first memory space to the second memory space of the slave cluster through remote direct memory access technology. There is no need for a TCP network to send data, which can reduce data transmission delays and CPU overhead, and improve data synchronization efficiency.

[0010] In one embodiment, obtaining target data includes:

[0011] receiving a business processing request, the business processing request including target data and control information;

[0012] The target data and the control information are decoupled to obtain the target data.

[0013] In the above embodiment, the target data and the control information are separated so that the target data does not need to be sent along with the control information. The target data and the control information can be transmitted in different ways, thereby improving the efficiency of target data synchronization.

[0014] In one embodiment, after synchronously writing the target data stored in the first memory space to the second memory space of the slave cluster based on remote direct memory access technology, the method further includes:

[0015] Determine data description information of the target data, where the data description information is used to represent storage information of the target data in the first memory space;

[0016] The data description information and the control information are sent to the slave cluster, where the data description information and the control information are used by the slave cluster to perform service processing according to the data description information and the control information to obtain a first service processing result.

[0017] In the above embodiment, by determining the data description information of the target data, when the master-slave cluster performs control flow information synchronization, only the data description information needs to be transmitted without transmitting the target data. This can reduce the CPU usage and network bandwidth usage caused by transmitting the target data between the master-slave clusters, and improve the efficiency of data synchronization.

[0018] In one embodiment, sending data description information and control information to a slave cluster includes:

[0019] The data description information and control information are sent to the network layer of the slave cluster through the network layer.

[0020] In one embodiment, the method further comprises:

[0021] Perform business processing according to the target data information to obtain a second business processing result;

[0022] After obtaining the second business processing result, obtaining the first business processing result;

[0023] A target business processing result corresponding to the business processing request is determined according to the first business processing result and the second business processing result.

[0024] In the above embodiment, the master cluster obtains the service processing results of the slave cluster after the service processing is completed, and then performs aggregation processing to obtain the target service processing results corresponding to the service processing request, thereby meeting the user's service processing needs.

[0025] In one embodiment, the data description information includes a memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing based on the target data in the second memory space and obtain a first business processing result.

[0026] In the above embodiment, the slave cluster determines the location of the target data in the second memory space based on the memory number and memory address information corresponding to the target data included in the data description information, thereby obtaining the target data. There is no need to carry the target data in the control flow, which avoids the target data following the heavy load flow of the control flow and improves the efficiency of data synchronization.

[0027] In one embodiment, the distributed storage system further includes a memory management node, which is configured to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0028] In the above embodiment, the memory management node manages the memory space of each cluster through the interaction between the main memory management agent module of the memory management node and the memory management agent modules run by the controllers of each cluster.

[0029] In one embodiment, the memory management node is used to apply for and update the first memory space, and the memory management node is also used to apply for and update the second memory space.

[0030] In one embodiment, the memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0031] In the above embodiment, when an abnormality occurs in the usage information of the first memory space and the second memory space, an alarm message is sent to the master cluster to implement monitoring of the first memory space and the second memory space.

[0032] In one embodiment, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0033] In the above embodiment, the service processing process of the master cluster and the service processing process of the slave cluster are performed asynchronously and in parallel, rather than synchronously and serially, thereby improving the efficiency of service processing.

[0034] In a second aspect, the present application further provides a data synchronization device, which is applied to a master cluster in a distributed storage system, wherein the distributed storage system includes a master cluster and a slave cluster, and the device includes:

[0035] The acquisition module is used to obtain target data, which is the data that the distributed storage system needs to process;

[0036] A storage module, configured to store the target data in the first memory space of the main cluster;

[0037] The synchronization module is used to synchronously write the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces.

[0038] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the data synchronization methods described in the first aspect when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data synchronization method described in any one of the first aspects above is implemented.

[0040] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data synchronization method described in any one of the first aspects above.

[0041] The above-mentioned data synchronization method, device, equipment, readable storage medium and program product are applied to the master cluster in the distributed storage system, which includes the master cluster and the slave cluster. The method includes: first, obtaining the target data that the distributed system needs to process, then storing the target data in the first memory space of the master cluster, and finally, based on the remote direct memory access technology, synchronously writing the target data stored in the first memory space to the same position in the second memory space of the slave cluster; wherein the first memory space and the second memory space are mirror memory spaces. In this way, when the master cluster and the slave cluster synchronize data, the master cluster saves the target data in the first memory space, and then writes the target data in the first memory space to the second memory space of the slave cluster through the remote direct memory access technology. There is no need for a TCP network to send data, which can reduce the delay of data transmission and CPU overhead, and the efficiency of data synchronization is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of the current master-slave cluster synchronization data flow in one embodiment;

[0044] Figure 2 This is a diagram of an application environment of a data synchronization method in one embodiment;

[0045] Figure 3 1 is a flow chart of a data synchronization method according to an embodiment;

[0046] Figure 4 A schematic diagram of a memory management node management mode in one embodiment;

[0047] Figure 5 Schematic diagram of a flow chart of the steps of sending data description information in one embodiment;

[0048] Figure 6 A flowchart of steps for determining a business processing result in one embodiment;

[0049] Figure 7 is a flowchart of a data synchronization method in another embodiment;

[0050] Figure 8 A data flow diagram of a data synchronization method in another embodiment;

[0051] Figure 9 is a structural block diagram of a data synchronization device in one embodiment;

[0052] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] During business operation, due to business needs, business data needs to be written to multiple clusters. For example, business data is written to both ends of the master and slave clusters. When a problem occurs in one of the clusters, the user business can use the other cluster to continue providing services, thereby ensuring the continuity of the user business.

[0055] For example, in disaster recovery scenarios, data synchronization methods for disaster recovery can be categorized into three types: asynchronous remote replication, synchronous remote replication, and active-active replication, based on the different RTO (Recovery Point Objective) and RPO (Recovery Time Objective) requirements of the business. Asynchronous remote replication has non-zero RTO and RPO, meaning that data will be lost for a period of time when switching between different clusters, and business recovery will take some time. Synchronous remote replication has a non-zero RTO and a zero RPO, meaning that data will not be lost when switching between different clusters, but it will take some time to recover from the outage. Active-active replication supports both read and write operations on both clusters, ensuring complete data synchronization between the two clusters. Therefore, it is more suitable for scenarios where both RTO and RPO are 0.

[0056] In a dual-active scenario, Figure 1 As shown, in the current conventional master-slave cluster data synchronization solution, the master cluster receives data from the business end and saves it to the data memory. After performing business processing on the local end, it copies the data to the network layer of the master cluster to obtain network data. The master cluster synchronizes the data to the slave cluster network receiving module via the TCP network. The slave cluster copies the data from the network receiving module to the data memory and performs business processing based on the data. It can be seen that the above process includes two copy processes between memory data and network data. At the same time, data synchronization between the master and slave clusters is performed through the TCP network, which requires a large amount of network bandwidth to transmit business data. At the same time, the data transmission link is long. Therefore, the efficiency of master-slave cluster data synchronization is not high.

[0057] In view of this, the present application proposes a data synchronization method, which is applied to a master cluster in a distributed storage system. The distributed storage system includes a master cluster and a slave cluster. When the master cluster and the slave cluster synchronize data, the master cluster saves the target data to a first memory space, and then writes the target data in the first memory space to the second memory space of the slave cluster through remote direct memory access technology. No TCP network is required to send data, which can reduce data transmission delays and CPU overhead, and data synchronization efficiency is higher.

[0058] The data synchronization method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown, the distributed system 20 includes a master cluster 202, a slave cluster 204, and a memory management node 206. The master cluster 202 communicates with the slave cluster 204 via a network, and the memory management node 206 is used to manage the memory on the master cluster 202 and the slave cluster 204. The master cluster 202 and the slave cluster 204 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or cloud servers providing cloud computing services.

[0059] In an exemplary embodiment, Figure 3 As shown, a data synchronization method is provided, which is applied to Figure 2 The main cluster in FIG is taken as an example to illustrate the process, including the following steps 301 to 303. Among them:

[0060] Step 301: Acquire target data.

[0061] Among them, the target data is the data that the distributed storage system needs to process. For example, the target data is determined based on the received business processing request. The business processing request can be the requirements for various business processing sent by the user, such as storage business or computing business. Optionally, taking the active-active scenario of the master cluster and the slave cluster as an example, the business processing process may include applying for memory, memory mirror synchronization, caching of business flow target data and metadata, persistent data writing, and waiting for summary results. Since the active-active method requires both the master cluster and the slave cluster to perform business processing and return the results after summarizing them, it is necessary to synchronize the target data required for business processing between the master cluster and the slave cluster. The target data can also be data that needs to be processed locally in the distributed storage system.

[0062] Step 302: Store the target data in the first memory space of the main cluster.

[0063] The first memory space can be a power-retention memory, a special type of memory that retains data in the event of a system power outage or failure, providing higher reliability and performance. The first memory space can be a portion of the memory in the primary cluster. The size of the first memory space can be set based on actual business needs and is not limited in this embodiment of the application.

[0064] Step 303: Synchronously write the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology.

[0065] The first memory space and the second memory space are mirrored memory spaces, i.e., they are identical in size and configuration, and logically correspond one-to-one. Remote Direct Memory Access (RDMA) is a highly efficient network communication technology that can directly transmit data between two nodes via a network interface card. This technology is faster than TCP network transmission. The master cluster can directly access the second memory space of the slave cluster without requiring processing by the operating system or CPU of the master or slave cluster. Specifically, the master cluster can write the target data directly to the same location in the second memory space of the slave cluster using a memory write command based on the target data's storage location in the first memory space. During the write process, the CPUs of the master and slave clusters do not need to participate in the processing. This significantly reduces data transmission latency and improves data synchronization efficiency by reducing CPU involvement and memory copy times.

[0066] In the above embodiment, a master cluster is applied to a distributed storage system, and the distributed storage system includes a master cluster and a slave cluster. The method includes: first, obtaining target data that the distributed system needs to process, then storing the target data in the first memory space of the master cluster, and finally, synchronously writing the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces. In this way, when the master cluster and the slave cluster synchronize data, the master cluster saves the target data in the first memory space, and then writes the target data in the first memory space to the second memory space of the slave cluster through remote direct memory access technology. There is no need for a TCP network to send data, which can reduce data transmission delays and CPU overhead, and improve data synchronization efficiency.

[0067] Optionally, obtaining the target data includes: receiving a business processing request, the business processing request including the target data and control information; and decoupling the target data from the control information to obtain the target data.

[0068] A business processing request sent by a user is received. The business processing request may include target data to be processed and control information during the processing. It can be understood that the target data has a large data volume and the control information has a small data volume. Therefore, the target data and the control information are decoupled, that is, the target data and the control information are separated to obtain the target data. The transmission methods of the target data and the control information can be different, so that the target data does not need to be sent along with the control information.

[0069] In one embodiment, Figure 2 As shown in , the distributed storage system further includes a memory management node, which is used to interact with the memory management agent on the master cluster and the memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0070] Among them, the power-saving memory is based on the MSEG (Memory Segment Manager) memory. MSEG is used to manage and allocate memory segments and is implemented based on the remote direct memory access technology of unilateral communication. Each cluster includes a controller, which is used to manage the resources within the cluster and monitor the status of the cluster. In order to manage the memory in each cluster, a memory management agent module runs in the controller in each cluster. The memory management node of the distributed storage system also includes a main memory management agent module. The main memory management agent module interacts with the memory management agent modules in each controller to manage the memory in each cluster. Figure 4 As shown, memory management includes MSEG memory application, MSEG memory update, collection of MSEG usage and acquisition of memory space usage information.

[0071] The memory management node is used for applying for and updating the first memory space, and the memory management node is also used for applying for and updating the second memory space.

[0072] Optionally, the MSEG memory application is mainly used to apply for the first memory space in the master cluster and the second memory space in the slave cluster. The MSEG memory update is mainly used to update the size and configuration of the first memory space in the master cluster and the second memory space in the slave cluster.

[0073] At the same time, the main memory management agent module can collect the memory usage of each cluster controller MSEG and the memory space usage information, so that users can check whether the memory usage of each cluster is consistent. If not, the data synchronization process may be abnormal.

[0074] The memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0075] Among them, the first preset threshold can be set according to the memory space of the main cluster, and the second preset threshold can be set according to the memory space of the slave cluster. The first preset threshold and the second preset threshold can be the same or different, and the embodiment of the present application does not limit this. After the memory management node obtains the usage information of the first memory space, if the usage information of the first memory space indicates that the space occupied by the first memory space has exceeded the first threshold, that is, the first memory space is too large, then an alarm message is sent to the main cluster, and the main cluster can display it to the user through an interface to facilitate the user to make timely adjustments. Optionally, the monitoring of the usage information of the second memory space of the slave cluster is the same as that of the first memory space.

[0076] In the above embodiment, the memory management node manages the memory space of each cluster through the interaction between the main memory management agent module of the memory management node and the memory management agent modules run by the controllers of each cluster.

[0077] In the embodiment of the present application, after synchronizing the target data of the master cluster to the slave cluster, as shown in FIG. Figure 5 As shown, the method further includes:

[0078] Step 501: Determine data description information of target data.

[0079] The data description information represents the storage information of the target data in the first memory space. Optionally, the data description information may include the memory location information where the target data is stored in the first memory space. This data description information can be used to locate the target data and perform business processing based on the target data. After the primary cluster stores the target data in the first memory space, it can obtain the data description information corresponding to the target data based on the storage location.

[0080] Optionally, the data description information may be a memory address or index information, and the memory location information is determined by the index information and the tree structure of the storage structure.

[0081] Step 502: Send data description information and control information to the slave cluster.

[0082] The data description information and control information are used by the slave cluster to perform business processing based on the data description information and the control information to obtain a first business processing result. After obtaining the data description information, the master cluster sends the data description information to the slave cluster through the control flow, that is, along with the control information. Because the data description information only includes the storage location of the target data and does not include the target data, this avoids the problem of directly transmitting the target data, which may consume a large amount of network bandwidth.

[0083] After receiving the data description information and control information, the slave cluster can determine the location of the target data in the first memory space through the data description information. Since the first memory space and the second memory space are mirror memories, that is, the location of the target data in the first memory space and the second memory space is the same, the location of the target data in the second memory space can be determined. The slave cluster can obtain the target data based on the location of the target data in the second memory space, perform business processing based on the target data and the control information, and obtain a first business processing result.

[0084] Optionally, the data description information and the control information are sent to the network layer of the slave cluster through the network layer, the data description information and the control information are constructed into network data, and the network data is sent to the network layer of the slave cluster through the network layer, such as the TCP network.

[0085] In the above embodiment, by determining the data description information of the target data, when the master-slave cluster performs control flow information synchronization, only the data description information needs to be transmitted without transmitting the target data. This can reduce the CPU usage and network bandwidth usage caused by transmitting the target data between the master-slave clusters, and improve the efficiency of data synchronization.

[0086] In an embodiment of the present application, the data description information includes the memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing according to the target data in the second memory space and obtain the first business processing result.

[0087] Optionally, the first memory space can be divided into multiple memory segments or memory pages of a fixed size, each memory segment or memory page corresponding to a memory number, and the data description information includes the memory number and memory address information corresponding to the target data. The slave cluster can determine the memory segment or memory page corresponding to the target data based on the memory number, and based on the determined memory segment or memory page and memory address information, it can determine the location of the target data in the second memory space, thereby obtaining the target data. After obtaining the target data, the slave cluster performs a business processing flow in parallel with the master cluster and obtains a first business processing result.

[0088] In the above embodiment, the slave cluster determines the location of the target data in the second memory space based on the memory number and memory address information corresponding to the target data included in the data description information, thereby obtaining the target data. There is no need to carry the target data in the control flow, which avoids the target data following the heavy load flow of the control flow and improves the efficiency of data synchronization.

[0089] In the embodiments of the present application, Figure 6 As shown, the method further includes:

[0090] Step 601: Perform business processing according to target data information to obtain a second business processing result.

[0091] Among them, the main cluster performs business processing based on the target data. The business processing process may include steps such as applying for memory, memory mirror synchronization, caching of business flow target data and metadata, and persistent data writing. After the processing is completed, the second business processing result is obtained. Optionally, the second business processing result may vary depending on the business processing requirements. For example, if the business processing requirement is to store data, the second business processing result may be information on successful data storage and the specific location of the data storage, or information on failed data storage and the reason for the storage failure. If the business processing requirement is to update data, the second business processing result may be information on successful data update or failed data update and the reason for the update failure.

[0092] Step 602: After obtaining the second business processing result, obtain the first business processing result.

[0093] After the primary cluster completes its business processing and obtains the second business processing result, it triggers the process of obtaining the first business processing result obtained by the secondary cluster. Optionally, an event can be used to trigger the secondary cluster to send the first business processing result to the primary cluster. If the secondary cluster's business processing process has not yet completed, the primary cluster waits for the secondary cluster's business processing process to complete until the secondary cluster's business processing is complete before sending the first business processing result to the primary cluster.

[0094] Step 603: Determine a target business processing result corresponding to the business processing request according to the first business processing result and the second business processing result.

[0095] In the active-active scenario of the master cluster and the slave cluster, after obtaining the first business processing result, the first business processing result and the second business processing result are aggregated to obtain the business processing result, and the business processing result is returned to the sender of the business processing request, that is, the business processing is completed.

[0096] In the above embodiment, the master cluster obtains the first business processing result of the slave cluster after completing the business processing, and then performs aggregation processing to obtain the target business processing result corresponding to the business processing request, thereby meeting the user's business processing needs.

[0097] Optionally, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0098] Currently, the master cluster usually first performs business processing based on the target data, then copies the target data to the network layer, and sends it to the slave cluster via the TCP network. The slave cluster then performs business processing based on the target data. In this way, the business processing process of the master cluster and the business processing process of the slave cluster are performed serially, and the slave cluster needs to process the data after the master cluster completes the processing. This business processing method is not efficient. Therefore, in this data synchronization method, the target data is synchronized from the first memory space to the second memory space using remote direct memory access technology. The control flow between the master cluster and the slave cluster only needs to transmit the data description information of the target data. Therefore, after the master cluster synchronizes the target data to the slave cluster, it sends the data description information of the target data, and then triggers the slave cluster to perform business processing. Business processing can also be performed within the master cluster. The business processing of the master cluster and the slave cluster can be asynchronous and parallel, and business processing can be performed based on the target data in their respective memory spaces at the same time, without the need for the master cluster to send the target data to the slave cluster after completing the business processing, and the slave cluster performs business processing serially.

[0099] In the above embodiment, the service processing process of the master cluster and the service processing process of the slave cluster are performed asynchronously and in parallel, rather than synchronously and serially, thereby improving the efficiency of service processing.

[0100] In the embodiments of the present application, Figure 7 The data synchronization method of the present application is shown in FIG, which specifically includes the following steps:

[0101] Step 701: Acquire target data.

[0102] Step 702: Store the target data in the first memory space of the main cluster.

[0103] Step 703: Synchronously write the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology.

[0104] Step 704: Determine the data description information of the target data.

[0105] Step 705: Send data description information and control information to the slave cluster.

[0106] Step 706: Perform business processing according to the target data information to obtain a second business processing result.

[0107] Step 707: After obtaining the second business processing result, obtain the first business processing result.

[0108] Step 708: Determine a target business processing result corresponding to the business processing request according to the first business processing result and the second business processing result.

[0109] In this embodiment, when the master cluster and the slave cluster synchronize data, the master cluster saves the target data to the first memory space, and then writes the target data in the first memory space to the second memory space of the slave cluster through remote direct memory access technology. There is no need for a TCP network to send data. By determining the data description information of the target data, when the master and slave clusters synchronize control flow information, only the data description information needs to be transmitted, and there is no need to transmit the target data. This can reduce the CPU usage and network bandwidth usage caused by transmitting target data between the master and slave clusters, and improve the efficiency of data synchronization.

[0110] In the embodiments of the present application, Figure 8 As shown, a data synchronization method is provided, which is described by taking the active-active scenario of the master cluster and the slave cluster as an example, and includes the following steps.

[0111] 1. The user's business flow is sent from cluster A to the interface processing layer of this cluster.

[0112] 2. Determine whether this cluster is the data synchronization master, that is, the primary cluster. The data synchronization master is the end that receives business data and initiates data synchronization.

[0113] 3. If yes, trigger the data synchronization process step 1 of this cluster.

[0114] 4. Data synchronization processing step 1 is used to synchronize the first memory space storing the target data of the cluster to cluster B through the MSEG synchronization mechanism. Cluster B is the slave cluster.

[0115] 5. Then, the master cluster A triggers the service IO module of the slave cluster B to perform asynchronous data processing of the slave cluster B.

[0116] 6. Step 1 of the data synchronization process for the active cluster A is completed, indicating that the target data synchronization is complete.

[0117] 7. The main cluster A performs business processing steps, which may include operations such as data persistence and metadata updates.

[0118] 8. After the business processing steps of the active cluster A are completed, the data synchronization processing step 2 is triggered.

[0119] 9. Data synchronization processing step 2 is used to obtain the first business processing result of the asynchronous execution from cluster B. If the processing is not completed, wait for the processing from cluster B to be completed and obtain the first business processing result.

[0120] 10. Finally, the second business processing result of the master cluster A and the first business processing result of the slave cluster B are summarized and returned to the user application layer, and the business processing flow ends.

[0121] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0122] Based on the same inventive concept, embodiments of the present application also provide a data synchronization device for implementing the aforementioned data synchronization method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more data synchronization device embodiments provided below can be found in the above-mentioned limitations of the data synchronization method and will not be repeated here.

[0123] In an exemplary embodiment, Figure 9 As shown, a data synchronization device 900 is provided, which is applied to a master cluster in a distributed storage system. The distributed storage system includes a master cluster and a slave cluster, including: an acquisition module 901, a storage module 902 and a synchronization module 903, wherein:

[0124] The acquisition module 901 is used to acquire target data, which is data that the distributed storage system needs to process;

[0125] A storage module 902 is configured to store the target data in a first memory space of the primary cluster;

[0126] The synchronization module 903 is configured to synchronously write the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces.

[0127] In one embodiment, the acquisition module is specifically configured to receive a service processing request including target data and control information; and decouple the target data from the control information to obtain the target data.

[0128] In one embodiment, the device also includes a sending module for determining data description information of the target data, the data description information being used to characterize storage information of the target data in the first memory space; sending the data description information and control information to the slave cluster, the data description information and control information being used for the slave cluster to perform business processing according to the data description information and control information to obtain a first business processing result.

[0129] In one embodiment, the sending module is specifically configured to send the data description information and the control information to the network layer of the slave cluster through the network layer.

[0130] In one embodiment, the device also includes a determination module for performing business processing based on the target data information to obtain a second business processing result; after obtaining the second business processing result, obtaining the first business processing result; and determining the target business processing result corresponding to the business processing request based on the first business processing result and the second business processing result.

[0131] In one embodiment, the data description information includes a memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing based on the target data in the second memory space and obtain a first business processing result.

[0132] In one embodiment, the distributed storage system further includes a memory management node, which is configured to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0133] In one embodiment, the memory management node is used to apply for and update the first memory space, and the memory management node is also used to apply for and update the second memory space.

[0134] In one embodiment, the memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0135] In one embodiment, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0136] Each module in the above-mentioned data synchronization device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store business data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0138] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0139] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: obtaining target data, where the target data is data that needs to be processed by a distributed storage system; storing the target data in a first memory space of a master cluster; and synchronously writing the target data stored in the first memory space to the same location in a second memory space of a slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirrored memory spaces.

[0140] In one embodiment, when executing the computer program, the processor further implements the following steps: receiving a service processing request, the service processing request including target data and control information; and decoupling the target data from the control information to obtain the target data.

[0141] In one embodiment, when the processor executes the computer program, it also implements the following steps: determining data description information of the target data, the data description information is used to characterize the storage information of the target data in the first memory space; sending the data description information and control information to the slave cluster, the data description information and control information are used for the slave cluster to perform business processing according to the data description information and control information to obtain a first business processing result.

[0142] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: sending the data description information and the control information to the network layer of the slave cluster through the network layer.

[0143] In one embodiment, when the processor executes the computer program, it also implements the following steps: performing business processing according to the target data information to obtain a second business processing result; after obtaining the second business processing result, obtaining the first business processing result; and determining the target business processing result corresponding to the business processing request based on the first business processing result and the second business processing result.

[0144] In one embodiment, the data description information includes the memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing based on the target data in the second memory space and obtain the first business processing result.

[0145] In one embodiment, the distributed storage system further includes a memory management node, which is configured to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0146] In one embodiment, the memory management node is used to apply for and update the first memory space, and the memory management node is also used to apply for and update the second memory space.

[0147] In one embodiment, the memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0148] In one embodiment, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining target data, where the target data is data that needs to be processed by a distributed storage system; storing the target data in a first memory space of a master cluster; and synchronously writing the target data stored in the first memory space to the same location in a second memory space of a slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirrored memory spaces.

[0150] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving a service processing request, the service processing request including target data and control information; and decoupling the target data from the control information to obtain the target data.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining data description information of the target data, the data description information is used to characterize the storage information of the target data in the first memory space; sending the data description information and control information to the slave cluster, the data description information and control information are used for the slave cluster to perform business processing according to the data description information and control information to obtain a first business processing result.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: sending the data description information and the control information to the network layer of the slave cluster through the network layer.

[0153] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing business processing according to the target data information to obtain a second business processing result; after obtaining the second business processing result, obtaining the first business processing result; and determining the target business processing result corresponding to the business processing request based on the first business processing result and the second business processing result.

[0154] In one embodiment, the data description information includes the memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing based on the target data in the second memory space and obtain the first business processing result.

[0155] In one embodiment, the distributed storage system further includes a memory management node, which is configured to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0156] In one embodiment, the memory management node is used to apply for and update the first memory space, and the memory management node is also used to apply for and update the second memory space.

[0157] In one embodiment, the memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0158] In one embodiment, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the following steps: obtaining target data, where the target data is data that needs to be processed by a distributed storage system; storing the target data in a first memory space of a master cluster; and synchronously writing the target data stored in the first memory space to the same location in a second memory space of a slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirrored memory spaces.

[0160] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving a service processing request, the service processing request including target data and control information; and decoupling the target data from the control information to obtain the target data.

[0161] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining data description information of the target data, the data description information is used to characterize the storage information of the target data in the first memory space; sending the data description information and control information to the slave cluster, the data description information and control information are used for the slave cluster to perform business processing according to the data description information and control information to obtain a first business processing result.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: sending the data description information and the control information to the network layer of the slave cluster through the network layer.

[0163] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing business processing according to the target data information to obtain a second business processing result; after obtaining the second business processing result, obtaining the first business processing result; and determining the target business processing result corresponding to the business processing request based on the first business processing result and the second business processing result.

[0164] In one embodiment, the data description information includes the memory number and memory address information corresponding to the target data. The memory number and memory address information are used by the cluster to search for the target data in the second memory space according to the memory number and memory address information, so as to perform business processing based on the target data in the second memory space and obtain the first business processing result.

[0165] In one embodiment, the distributed storage system further includes a memory management node, which is configured to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

[0166] In one embodiment, the memory management node is used to apply for and update the first memory space, and the memory management node is also used to apply for and update the second memory space.

[0167] In one embodiment, the memory management node is also used to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; the memory management node is also used to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

[0168] In one embodiment, the business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0170] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0171] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0172] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data synchronization method, characterized in that: Applied to a master cluster in a distributed storage system, the distributed storage system including the master cluster and slave clusters, the method comprising: Acquire target data, where the target data is data that the distributed storage system needs to process; storing the target data in the first memory space of the main cluster; The target data stored in the first memory space is synchronously written to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces.

2. The method according to claim 1, characterized in that The acquiring target data includes: receiving a service processing request, wherein the service processing request includes target data and control information; The target data and the control information are decoupled to obtain the target data.

3. The method according to claim 2, characterized in that After synchronously writing the target data stored in the first memory space to the second memory space of the slave cluster based on remote direct memory access technology, the method further includes: Determining data description information of the target data, where the data description information is used to represent storage information of the target data in the first memory space; The data description information and the control information are sent to the slave cluster, where the data description information and the control information are used by the slave cluster to perform service processing according to the data description information and the control information to obtain a first service processing result.

4. The method according to claim 3, characterized in that The sending the data description information and the control information to the slave cluster includes: The data description information and the control information are sent to the network layer of the slave cluster through the network layer.

5. The method according to claim 4, characterized in that The method further comprises: Performing business processing according to the target data information to obtain a second business processing result; After obtaining the second business processing result, obtaining the first business processing result; A target business processing result corresponding to the business processing request is determined according to the first business processing result and the second business processing result.

6. The method according to claim 3, characterized in that The data description information includes the memory number and memory address information corresponding to the target data. The memory number and the memory address information are used by the slave cluster to search for the target data in the second memory space according to the memory number and the memory address information, so as to perform business processing based on the target data in the second memory space to obtain the first business processing result.

7. The method according to claim 1, characterized in that The distributed storage system further includes a memory management node, which is used to interact with a memory management agent on the master cluster and a memory management agent on the slave cluster to manage the memory space of the master cluster and the slave cluster.

8. The method according to claim 7, characterized in that The memory management node is used for applying for and updating the first memory space, and the memory management node is also used for applying for and updating the second memory space.

9. The method according to claim 8, characterized in that The memory management node is further configured to obtain usage information of the first memory space, and send an alarm message to the main cluster when the usage information of the first memory space exceeds a first preset threshold; The memory management node is further configured to obtain usage information of the second memory space, and send an alarm message to the main cluster when the usage information of the second memory space exceeds a second preset threshold.

10. The method according to claim 3, characterized in that The business processing of the master cluster and the business processing of the slave cluster are performed asynchronously and in parallel.

11. A data synchronization device, characterized in that: A device applied to a master cluster in a distributed storage system, wherein the distributed storage system includes the master cluster and a slave cluster, includes: An acquisition module, configured to acquire target data, wherein the target data is data that needs to be processed by the distributed storage system; A storage module, configured to store the target data in the first memory space of the master cluster; A synchronization module is used to synchronously write the target data stored in the first memory space to the same location in the second memory space of the slave cluster based on remote direct memory access technology; wherein the first memory space and the second memory space are mirror memory spaces.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images