Data synchronization method and device, equipment and medium

Abstract

The invention relates to the technical field of computers, and discloses a data synchronization method, device and equipment and a medium, and the method comprises the steps that an old storage service system adds metadata of historical data into a shared storage pool according to a first data synchronization request, and adds metadata of data corresponding to a newly obtained I / O request into a message queue, and sending a first response message to the new storage service system to instruct the new storage service system to read the metadata from the shared storage pool and then execute a data synchronization operation, and adding the metadata in the message queue into the shared storage pool according to the second data synchronization request, and a second response message is sent to the new storage service system to instruct the new storage service system to read the remaining metadata from the shared storage pool and then execute the data synchronization operation again, so that the new storage service system can acquire the metadata of the historical data and can timely synchronize the metadata of the newly added data. Seamless and rapid switching of business service from an old system to a new storage service system is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data synchronization method, device, equipment and medium. Background Art

[0002] A distributed storage system is a distributed system, that is, multiple nodes provide unified services in a cluster working mode. The key issue of upgrading a distributed storage system is that a multi-copy strategy is usually adopted in a distributed storage system to ensure data reliability, and a strong consistency model is adopted to ensure the consistency of the copies. The strong consistency model is that when one of the multiple nodes receives an I / O request, the processing of the I / O request will be immediately synchronized to all nodes, keeping the data copies on all nodes consistent at all times.

[0003] The key to the strong consistency model is that when a node receives an I / O request, the I / O will only return successfully after all replicas of all nodes have completed the I / O processing. This will cause the storage service system to restart on the upgraded node during the storage service system upgrade, causing the I / O requests issued by the upper-layer business client to wait for the I / O return from the upgraded node during the restart period and the time period when the service can be provided again. If the upgraded node does not return for a long time, the upper-layer business client will receive prompts such as no I / O return or return timeout, and the performance test will show that the write I / O is 0 for a short time. Summary of the invention

[0004] In view of this, the present invention provides a data synchronization method, device, equipment and medium to solve the problem that when a storage service system of any node in a distributed storage system is upgraded, the system is restarted, resulting in an inability to process I / O requests and causing the entire distributed storage system to have no return on I / O requests.

[0005] In a first aspect, the present invention provides a data synchronization method, which is applied to a distributed storage system, the distributed storage system includes multiple nodes for performing upgrade services, each node includes an old storage service system and an upgraded new storage service system, the method is executed by the old storage service system in a first node, the first node is any one of the multiple nodes, and the method includes:

[0006] Receiving a first data synchronization request sent by the new storage service system;

[0007] According to the first data synchronization request, metadata of historical data stored in the old storage service system is added to a pre-built shared storage pool, and metadata of data corresponding to the newly obtained I / O request is added to a pre-built message queue;

[0008] Sending a first response message to the new storage service system, where the first response message is used to instruct the new storage service system to perform a metadata synchronization operation on the new storage service system after reading the metadata of the historical data from the shared storage pool;

[0009] Obtaining a second data synchronization request sent by the new storage service system;

[0010] According to the second data synchronization request, the metadata in the message queue is added to the shared storage pool, and a second response message is sent to the new storage service system. The second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool and then perform the metadata synchronization operation on the new storage service system again.

[0011] The data synchronization method provided by the present invention has the following advantages:

[0012] The method maintains the business service of the old storage service system at the front end, and the new storage service system at the back end sends a first data synchronization request. The old storage service system adds the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and then feeds back a first response message. It ensures that the new storage service system can obtain the metadata information of the data accumulated in the old system for a long time. The new storage service system sends a second data synchronization request, and the old storage service system adds the metadata in the message queue to the shared storage pool, and sends a second response message to the new storage service system. It ensures that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner. The two data synchronization requests and response messages ensure that the new storage service system can not only obtain the metadata of the historical data, but also synchronize the metadata of the newly added data in a timely manner. In addition, the data synchronization process does not interrupt the reception and processing of the I / O request, so that the front-end business interface of the data synchronization process is not perceived.

[0013] During data synchronization, the processing of the I / O strong consistency model continues to be handled by the entire old cluster, that is, after any node receives an I / O request from the client, it will be synchronized to the remaining nodes in the old cluster and processed by the old storage service system. After data synchronization stops, the new cluster takes over, and the I / O request is processed by the new storage service system of each node. This method cleverly avoids the restart step during the upgrade of the old storage service system to the new storage service system, and does not interrupt the processing of the client's I / O requests, realizing seamless and rapid switching of business services from the old storage service system to the new storage service system.

[0014] In an optional implementation, the distributed storage system includes multiple nodes for executing the upgrade service, the multiple nodes include an upgrade service control node, and before receiving the first data synchronization request sent by the new storage service system, the method further includes:

[0015] Receive a data synchronization instruction issued by the upgrade service control node;

[0016] Create a shared storage pool according to the data synchronization instructions.

[0017] Specifically, according to the data synchronization instructions issued by the upgrade service control node, all nodes proceed simultaneously, saving a lot of time, reducing various uncertainties in the upgrade process, improving the stability of the cluster, and creating a shared storage pool to provide a unified metadata access entry for the new storage service system. The new storage service system does not need to search and obtain metadata in the old storage service system, but only needs to read from the shared storage pool, which simplifies the metadata acquisition process and improves metadata acquisition efficiency.

[0018] In an optional implementation, before receiving the first data synchronization request sent by the new storage service system, the method further includes:

[0019] Receiving a communication connection link creation request sent by the new storage service system;

[0020] According to the communication connection link creation request, a communication connection link is created between the new storage service system and the new storage service system so that the new storage service system sends a data synchronization request through the communication connection link, wherein the data synchronization request includes a first data synchronization request and a second data synchronization request.

[0021] Specifically, the communication connection link can ensure that data is transmitted and received in the order in which it is sent. For example, in the data synchronization process described above, the sending and receiving of the first data synchronization request and the first response message, and the sending and receiving of the second data synchronization request and the second response message are in sequence. By establishing a communication connection link, the sequence of these messages can be ensured, and synchronization errors caused by data disorder can be avoided, thereby ensuring the accuracy of data synchronization.

[0022] In addition, the communication link usually has an error detection mechanism, such as a checksum, which can detect whether errors occur during data transmission. Once an error is found, it can be recovered through mechanisms such as retransmission to ensure the integrity and accuracy of the data.

[0023] In an optional implementation, before adding the metadata in the message queue to the shared storage pool according to the second data synchronization request, the method further includes:

[0024] Determine the amount of metadata in the message queue;

[0025] When the amount of metadata in the message queue is less than a first preset amount threshold, sending notification information to the upgrade service control node, the notification information is used to indicate that the amount of remaining metadata in the message queue is less than or equal to the first preset amount threshold;

[0026] Receive a stop data synchronization message sent by the upgrade service control node, where the stop data synchronization message is broadcasted to all nodes by the upgrade service control node after receiving notification information fed back by all nodes respectively;

[0027] According to the stop data synchronization message, stop the business services in the old storage service system.

[0028] Specifically, by determining the amount of metadata in the message queue and comparing it with the first preset amount threshold, whether to send notification information is determined, and the timing of stopping the service is dynamically determined according to the actual remaining amount of metadata, so as to avoid stopping the service too early when there is still a lot of metadata, resulting in incomplete data synchronization.

[0029] Each node independently determines and sends notification information based on the amount of metadata in its own message queue. The upgrade service control node broadcasts the stop data synchronization message only after receiving notification information from all nodes. The distributed collaborative decision-making mechanism ensures the comprehensiveness and accuracy of the business stop decision. Only when all nodes meet the conditions for stopping the business, the entire cluster will stop the business uniformly, avoiding inconsistent cluster status caused by incomplete data synchronization of some nodes, which affects the processing of the I / O strong consistency model.

[0030] In an optional implementation, after stopping the business service in the old storage service system according to the stop data synchronization message, the method further includes:

[0031] Detect the amount of metadata in the message queue, and send an end instruction to the new storage service system until the amount of metadata in the message queue is less than or equal to a second preset number threshold, wherein the end instruction is used to instruct the new storage service system to stop obtaining metadata from the shared storage pool, and the second preset number threshold is less than the first preset number threshold.

[0032] Specifically, the end instruction is sent only when the amount of metadata in the message queue is detected to be less than or equal to the second preset amount threshold, ensuring that all metadata in the old storage service system has been transferred to the shared storage pool. The new storage service system stops obtaining metadata from the shared storage pool according to the end instruction. The above mechanism can prevent the new storage service system from performing unnecessary data acquisition operations after the data has been completely synchronized.

[0033] And the end instruction can avoid the new storage service system from sending data synchronization requests to the old storage service system and wasting system resources.

[0034] In an optional implementation, the method is applied to a distributed storage system, the distributed storage system includes multiple nodes for performing upgrade services, each node includes an old storage service system and an upgraded new storage service system, the method is performed by the new storage service system in a first node, the first node is any one of the multiple nodes, and the method includes:

[0035] Sending a first data synchronization request to the old storage service system;

[0036] Receive a first response message fed back by the old storage service system, where the first response message is a response message sent by the old storage service system after adding metadata of historical data stored in the old storage service system to a pre-built shared storage pool according to the first data synchronization request;

[0037] After reading the metadata of the historical data from the shared storage pool according to the first response message, a metadata synchronization operation is performed;

[0038] After performing the metadata synchronization operation, a second data synchronization request is sent to the old storage service system;

[0039] Receive a second response message fed back by the old storage service system, where the second response message is a response message fed back by the old storage service system after adding the metadata in the message queue to the shared storage pool according to the second data synchronization request;

[0040] According to the second response message, the metadata synchronization operation is performed again after reading the remaining metadata from the shared storage pool.

[0041] Specifically, the new storage service system sends a first data synchronization request to synchronize the metadata of historical data stored in the old storage service system. This ensures that the new storage service system can obtain the metadata information of the data accumulated in the old storage service system over a long period of time, providing a basis for subsequent data management and query operations. A second data synchronization request is sent to synchronize the newly added metadata in the message queue, ensuring that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner, ensuring the integrity of the metadata. The response message provides the new storage service system with clear data synchronization status information, and the new storage service system performs subsequent operations based on the response message, avoiding the new storage service system from constantly monitoring whether the old storage service system stores all metadata in the shared storage pool, resulting in a waste of resources.

[0042] In an optional embodiment, the method further includes:

[0043] Receive an end instruction sent by the old storage service system;

[0044] According to the end instruction, stop obtaining metadata from the shared storage pool;

[0045] Start the business service and establish a session connection with the business service client.

[0046] Specifically, the old storage service system will send an end instruction only when it has no metadata to transfer, ensuring that the metadata is completely transferred. According to the end instruction, the new storage service system stops obtaining metadata from the shared storage pool, starts business services, and establishes a session connection with the new storage service system. This orderly switching process ensures that the business service client completes the migration from the old storage service system to the new storage service system with almost no perception, ensuring business continuity. In addition, all nodes perform system upgrades at the same time, which also avoids considering software version compatibility issues in the old and new storage service systems.

[0047] In a second aspect, the present invention provides a data synchronization device, the device comprising:

[0048] A receiving module, used for receiving a first data synchronization request sent by the new storage service system;

[0049] The transfer module is used to add the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and to add the metadata of the data corresponding to the newly obtained I / O request to the pre-built message queue;

[0050] A sending module, used for sending a first response message to the new storage service system, where the first response message is used for instructing the new storage service system to perform a data synchronization operation on the new storage service system after reading metadata of historical data from the shared storage pool;

[0051] The receiving module is further used to obtain a second data synchronization request sent by the new storage service system;

[0052] The transfer module is further used to add the metadata in the message queue to the shared storage pool according to the second data synchronization request;

[0053] The sending module is also used to send a second response message to the new storage service system, where the second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool and then perform a data synchronization operation on the new storage service system again.

[0054] A data synchronization device provided by the present invention has the following advantages: by maintaining the business service of the old storage service system at the front end, the new storage service system at the back end sends a first data synchronization request, and the old storage service system adds the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and then feeds back a first response message. It ensures that the new storage service system can obtain the metadata information of the data accumulated in the old system for a long time. The new storage service system sends a second data synchronization request, and the old storage service system adds the metadata in the message queue to the shared storage pool, and sends a second response message to the new storage service system. It ensures that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner. The two data synchronization requests and response messages ensure that the new storage service system can not only obtain the metadata of the historical data, but also synchronize the metadata of the newly added data in a timely manner. In addition, the data synchronization process does not interrupt the reception and processing of the I / O request, so that the front-end business interface of the data synchronization process is not perceived.

[0055] During data synchronization, the processing of the I / O strong consistency model continues to be handled by the entire old cluster, that is, after any node receives an I / O request from the client, it will be synchronized to the remaining nodes in the old cluster and processed by the old storage service system. After data synchronization stops, the new cluster takes over, and the I / O request is processed by the new storage service system of each node. This method cleverly avoids the restart step during the upgrade of the old storage service system to the new storage service system, and does not interrupt the processing of the client's I / O requests, realizing seamless and rapid switching of business services from the old storage service system to the new storage service system.

[0056] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data synchronization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0057] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data synchronization method of the first aspect or any corresponding embodiment thereof.

[0058] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data synchronization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a structural diagram of an offline upgrade method of a distributed storage system;

[0061] Figure 2 It is a flowchart of a data synchronization method provided by an embodiment of the present invention;

[0062] Figure 3 is a flow chart of another data synchronization method provided by an embodiment of the present invention;

[0063] Figure 4 is a flowchart of another data synchronization method provided by an embodiment of the present invention;

[0064] Figure 5 It is a structural diagram of a distributed storage system online upgrade method provided by an embodiment of the present invention;

[0065] Figure 6 It is a structural schematic diagram of the data synchronization process of each node in the distributed storage system cluster provided by an embodiment of the present invention;

[0066] Figure 7 is a structural block diagram of a data synchronization device provided by an embodiment of the present invention;

[0067] Figure 8 It is a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] The present invention relates to an online upgrade method for a distributed storage system in the technical field of cloud computing data centers. The distributed storage system is firstly a distributed system, that is, multiple nodes provide unified services in a cluster working mode. The upgrade process of the distributed storage system can be divided into online upgrade and offline upgrade. For the offline upgrade process, such as Figure 1 The following is a method for offline upgrade of a distributed storage system. First, check the system environment before upgrading to ensure that the current system operating environment supports offline upgrades. Next, update the software packages of multiple nodes in the distributed storage system. After completing the offline upgrade as prompted, restart the upgrade node service. Finally, wait for the node to come online before the upgrade is complete.

[0070] This process requires all businesses running on the distributed storage system to be suspended. Therefore, the demand for online upgrades in distributed storage systems is relatively widespread.

[0071] The key issue of online upgrade is that in order to ensure data reliability, a multi-copy strategy is usually adopted in distributed storage systems. At the same time, a strong consistency model is adopted to ensure the consistency of copies. That is, when all copies have completed I / O processing, the I / O returns success. For the online upgrade scenario, in order to make the upgraded system effective, some services need to be restarted, including services for data storage. However, the restart of the storage service on the upgraded node will cause the I / O request to wait for the I / O return of the upgraded node during the restart period and the time period when the service can be provided again. The strong consistency model causes the I / O request to keep waiting for the I / O return of the upgraded node. The performance of this process to the upper-level business is that the I / O has no return or returns timeout. The performance during the performance test is that the write I / O is 0 for a short time.

[0072] To solve the above problems, an embodiment of the present invention provides a data synchronization embodiment. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system (computer device) including a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] Before introducing the data synchronization method of the embodiment of the present application, it is necessary to first introduce the following content: install the new storage service system in each node and build a cluster corresponding to the new storage service system, and do the preparation work before data synchronization. The preparation work is performed by the old storage service system in all nodes in the distributed storage system, and the preparation work specifically includes:

[0074] Step a1: Acquire a first broadcast message sent by an upgrade service control node.

[0075] Specifically, as described above, the distributed storage system includes multiple nodes that perform upgrade services. After the distributed storage system starts the upgrade process, it is necessary to first select a node from the nodes to be upgraded as the upgrade service control node, which is responsible for controlling and managing the entire cluster upgrade process.

[0076] Step a2: Detect the installation environment corresponding to the new storage service system according to the first broadcast message.

[0077] Specifically, all nodes in the distributed storage system obtain a first broadcast message sent by the upgrade service control node, where the first broadcast message is used to instruct each node to detect whether it meets preset requirements for installing a new storage service system.

[0078] Among them, the inspection items include but are not limited to the following items: check the system resource usage, including CPU and memory, to ensure sufficient resources to meet the installation of the storage service system to be upgraded; check the stability of the network connection to ensure that data transmission will not be interrupted during the upgrade process; check the compatibility of the old storage service system with the new storage service system, including hardware and software; back up important data to prevent data loss due to unexpected failures during the upgrade process.

[0079] Step a3: Feedback the detection result to the upgrade service control node.

[0080] Specifically, no matter whether each node detects that it meets the preset requirements for installing a new storage service system, it needs to feed back the detection results to the upgrade service control node. When a node in the distributed storage system detects that it does not meet the preset requirements for installing a new storage service system, the entire distributed storage system will suspend the upgrade process. When all nodes in the distributed storage system meet the preset requirements for installing a new storage service system, the upgrade service control node determines that the installation environment of all nodes meets the preset requirements based on the detection results fed back by all nodes, and then sends installation instruction information.

[0081] Step a4: receiving installation instruction information sent by the upgrade service control node.

[0082] Step a5: According to the installation instruction information, obtain the upgrade package corresponding to the storage service system to be upgraded from the pre-built image platform.

[0083] Specifically, after receiving the installation instruction information sent by the upgrade service control node, each node obtains an upgrade package corresponding to the new storage service system from the pre-built image platform, wherein the upgrade package includes the installation package of the new storage service system and related configuration files.

[0084] Step a6: Install the new storage service system according to the upgrade package.

[0085] Specifically, when each node installs the new storage service system according to the upgrade package, it is necessary to ensure that all relevant files of the new storage service system and all relevant files of the old storage service system are in different directories to avoid conflicts between the new and old service systems.

[0086] Step a7: Feedback the installation result to the upgrade service control node.

[0087] Specifically, each node performs the installation of the new storage service system, and whether the installation is successful or not, it needs to feed back the installation result to the upgrade service control node. The upgrade service control node determines the content of the broadcast message to be sent next based on the installation result fed back by each node.

[0088] Step a8: receiving a second broadcast message sent by the upgrade service control node, and performing installation rollback or building a cluster corresponding to the new storage service system according to the second broadcast message instructions.

[0089] Specifically, the upgrade service control node detects whether the installation of the new storage service system on each node is successful based on the installation results fed back by each node. When a node installation fails in the distributed storage system, the upgrade service control node sends a second broadcast message to instruct each node to roll back the installation of the new storage service system. After receiving this broadcast message, each node needs to clean up the related files of the installed new storage service system to ensure that the installation progress of all nodes is synchronized.

[0090] When all nodes in the distributed storage system are successfully installed, the upgrade service control node determines that all nodes are successfully installed according to the installation results fed back by all nodes, and then sends a second broadcast message to instruct each node to build a cluster corresponding to the new storage service system.

[0091] The clusters corresponding to building the new storage service system include:

[0092] Step b1: After starting the new storage service system, the new storage service system receives the disk usage permission given by the old storage service system.

[0093] Step b2: The new storage service system loads the disk normally while ensuring that the business service with the client is closed.

[0094] Specifically, the data processing corresponding to each I / O request will be written to the disk of the old storage service system, and the metadata of the data corresponding to the I / O request is stored locally in the old storage service system. The metadata records the basic properties of the data, including the storage location of the data on the disk. Therefore, after granting the new storage service system permission to use the disk, the new storage service system only needs to receive the metadata corresponding to the data to operate the data on the disk, reducing the risk of data loss during transfer. Moreover, metadata occupies less memory than data and has a faster transmission rate. Based on this feature, the subsequent data synchronization method transmits the metadata of the data rather than the data itself.

[0095] The client and the storage service system establish a session connection through the Internet Small Computer System Interface (ISCSI) service. At this point, the data synchronization process has not yet started, the new storage service system does not have the metadata corresponding to the data, and cannot perform any operations on the data in the disk. Therefore, it is necessary to ensure that the business service with the client is closed, and the old storage service system continues to perform the business service with the client.

[0096] Step b3: Use new ports to establish communication connection links between the new storage service systems of each node.

[0097] Specifically, the communication connection links between nodes are used to ensure the normal execution of information interaction and multi-copy strong consistency model between nodes. The new ports are used to establish communication connection links between new storage service systems. This can not only ensure the normal execution of multi-copy strong consistency model of the old cluster during data synchronization, but also facilitate the normal execution of multi-copy strong consistency model of the new cluster after the new storage service system is started. Among them, the multi-copy strong consistency model is that when a node executes an I / O process, all copies have completed the same I / O process, and the I / O returns success.

[0098] After the requirements of the above three steps are met, the cluster corresponding to the new storage service system is successfully constructed. After the cluster corresponding to the new storage service system is successfully constructed, the cluster status corresponding to the new storage service system is set to ready. After that, the following method steps of this application are executed.

[0099] In this embodiment, a data synchronization method is provided, which can be used for the above-mentioned terminal devices, such as mobile phones, tablet computers, etc. The method is applied to a distributed storage system, which includes multiple nodes for performing upgrade services, each of which includes an old storage service system and an upgraded new storage service system, and the method is performed by the old storage service system in a first node, wherein the first node is any one of the multiple nodes, Figure 2 is a flow chart of a data synchronization method provided by an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0100] Step S201: receiving a first data synchronization request sent by a new storage service system.

[0101] Specifically, after the upgraded service control node detects that the new cluster is ready, it broadcasts a command carrying a data synchronization start message. The new storage service system in each node sends a first data synchronization request to the old storage service system according to the data synchronization start message, and the old storage service system starts the data synchronization operation after receiving the first data synchronization request.

[0102] Step S202: adding metadata of historical data stored in the old storage service system to a pre-built shared storage pool according to the first data synchronization request, and adding metadata of data corresponding to the newly acquired I / O request to a pre-built message queue.

[0103] Specifically, as mentioned above, the historical data stored in the disk and its corresponding metadata are already in a stable state. The old storage service system obtains the metadata of the historical data stored in the disk and adds it to the pre-built shared storage pool according to the pre-built format requirements, so that the metadata of the historical data can be synchronized to the new storage service system with the fastest reading efficiency. During the data synchronization process, new I / O requests may be continuously generated, and the data corresponding to the newly obtained I / O requests may still be in the process of processing. This part of the metadata is continuously increasing, so it is necessary to pre-build a message queue to store the metadata corresponding to the data stored in the disk after the data synchronization is turned on, so as to determine the time to send the data synchronization stop instruction based on the amount of metadata in the queue.

[0104] Step S203: Send a first response message to the new storage service system.

[0105] Specifically, after adding the metadata of the historical data to the pre-built shared storage pool, a response message is sent to the new storage service system to instruct the new storage service system to read the metadata of the historical data from the shared storage pool and then update the metadata to the new storage service system.

[0106] Step S204: Obtain a second data synchronization request sent by the new storage service system.

[0107] Specifically, after the new storage service system stores the metadata corresponding to the historical data in the new storage service system, it sends a second data synchronization request to the old storage service system, requesting to obtain the metadata in the message queue.

[0108] Step S205: According to the second data synchronization request, the metadata in the message queue is added to the shared storage pool, and a second response message is sent to the new storage service system.

[0109] Specifically, after receiving the second data synchronization request, the old storage service system adds the metadata in the message queue to the shared storage pool and sends a second response message to the new storage service system. The second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool. After the new storage system reads the above metadata, it can access the data stored in the disk through the metadata to achieve data synchronization.

[0110] The data synchronization method provided in this embodiment maintains the business service of the old storage service system at the front end, and the new storage service system at the back end sends a first data synchronization request. The old storage service system adds the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and then feeds back a first response message. It ensures that the new storage service system can obtain the metadata information of the data accumulated in the old system for a long time. The new storage service system sends a second data synchronization request, and the old storage service system adds the metadata in the message queue to the shared storage pool, and sends a second response message to the new storage service system. It ensures that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner. The two data synchronization requests and response messages ensure that the new storage service system can not only obtain the metadata of the historical data, but also synchronize the metadata of the newly added data in a timely manner. In addition, the data synchronization process does not interrupt the reception and processing of the I / O request, so that the front-end business interface of the data synchronization process is not perceived.

[0111] During the data synchronization process, the processing of the I / O strong consistency model continues to be handled by the entire old cluster, and is taken over by the new cluster after the data synchronization stops. Since the data synchronization process does not interrupt the reception and processing of I / O requests, there is no need to restart some services. This method cleverly avoids considering the processing of the I / O strong consistency model and realizes seamless and rapid switching of business services from the old storage service system to the new storage service system.

[0112] On the basis of the above-mentioned embodiment, whether it is historical data before data synchronization is enabled or new data obtained after data synchronization is enabled, the old storage service system needs to store its corresponding metadata in the shared storage pool first, and then send a response message to instruct the new storage service system to obtain it from the shared storage pool. In an optional embodiment, before receiving the first data synchronization request sent by the new storage service system, the process includes the following steps:

[0113] Step c1, receiving a data synchronization instruction sent by the upgrade service control node.

[0114] The data synchronization instruction is used to instruct each node to start data synchronization operations respectively.

[0115] Step c2: Create a shared storage pool according to the data synchronization instruction.

[0116] Specifically, as described above, after the cluster corresponding to the new storage service system is successfully built, the cluster status corresponding to the new storage service system is set to ready. After the upgraded service control node detects that the new cluster status is ready, it broadcasts a data synchronization start instruction. After receiving the data synchronization start instruction, the old storage service system creates a shared storage pool.

[0117] Specifically, the metadata of historical data stored in the old storage service system and the metadata of data corresponding to newly obtained I / O requests are centrally stored in a shared storage pool, so that the metadata in the old storage service system that was originally scattered in different locations can be gathered together, providing a unified metadata access portal for the new storage service system. The new storage service system does not need to search and obtain metadata in the old storage service system, but only needs to read it from the shared storage pool, which simplifies the metadata acquisition process and improves the efficiency of metadata acquisition.

[0118] On the basis of the above-mentioned embodiment, the sending and receiving of data synchronization request and response messages between the new and old storage service systems must rely on the corresponding communication connection link. Specifically, in an optional embodiment, before receiving the first data synchronization request sent by the new storage service system, the establishment of the communication connection link includes the following steps:

[0119] Step d1: receiving a communication connection link creation request sent by a new storage service system.

[0120] Specifically, the new storage service system sends a request to the old storage service system to create a communication connection link, so that the new storage service system sends a data synchronization request through the communication connection link to obtain metadata in the old storage service system, wherein the communication connection link can be a Transmission Control Protocol (TCP for short).

[0121] Step d2: Create a communication connection link with the new storage service system according to the communication connection link creation request.

[0122] Specifically, the old storage service system creates a communication connection link with the new storage service system according to the communication connection link creation request, and subsequently receives data synchronization requests and sends response messages through the communication connection link.

[0123] The data synchronization request includes a first data synchronization request and a second data synchronization request, and the response message also includes a first response message and a second response message.

[0124] Specifically, the communication connection link can ensure that data is transmitted and received in the order in which it is sent. For example, during the data synchronization process, the sending and receiving of the first data synchronization request and the first response message, and the sending and receiving of the second data synchronization request and the second response message are in sequence. By establishing a communication connection link, the sequence of these messages can be ensured, and synchronization errors caused by data disorder can be avoided, thereby ensuring the accuracy of data synchronization.

[0125] In addition, the communication link usually has an error detection mechanism, such as a checksum, which can detect whether errors occur during data transmission. Once an error is found, it can be recovered through mechanisms such as retransmission to ensure the integrity and accuracy of the data.

[0126] Based on any of the foregoing embodiments, since metadata occupies less memory and has a faster transmission rate than the data itself, in the old storage system, the rate of transferring metadata in the message queue to the shared storage pool is much greater than the rate of storing metadata in the message queue. Therefore, the amount of metadata in the message queue is constantly decreasing, and the metadata transmission rate after it decreases to below the preset quantity threshold is extremely fast and can be ignored. When the amount of metadata remaining in the message queues of the old storage service systems of all nodes is less than the preset quantity threshold, all storage service systems of the old cluster stop business services, and all new storage services of the new cluster start business services.

[0127] Specifically, in an optional embodiment, it can be used in the above-mentioned mobile terminal, such as a mobile phone, a tablet computer, etc. Figure 3 is a flow chart of a data synchronization method provided by an embodiment of the present invention, such as Figure 3 As shown, according to the second data synchronization request, before adding the metadata in the message queue to the shared storage pool, the step of each node stopping the business service in the old storage service system according to the amount of metadata in the message queue includes:

[0128] Step S301, determining the amount of metadata in the message queue.

[0129] Specifically, as described above, the second data synchronization request is used to indicate that the metadata in the message queue is added to the shared storage pool. After the old storage service system receives the second data synchronization request, it needs to first determine the amount of metadata in the message queue, and determine whether it is necessary to send a notification message to the upgraded service control node based on the amount of metadata in the message queue and the first preset number threshold.

[0130] Step S302: When the amount of metadata in the message queue is less than a first preset amount threshold, a notification message is sent to the upgrade service control node.

[0131] Step S303: receiving a stop data synchronization message sent by the upgrade service control node.

[0132] Specifically, when the amount of metadata in the message queue is less than a first preset threshold (e.g., 5 metadata), as described above, at this time, since the first preset threshold is small, the metadata transmission rate below the first preset threshold is almost imperceptible to the client. The old storage service system sends a notification message to the upgrade service control node, and the notification message is used to indicate that the amount of remaining metadata in the message queue is less than or equal to the first preset threshold.

[0133] After receiving the notification information fed back by all nodes respectively, the upgrade service control node broadcasts a message to stop data synchronization.

[0134] Step S304: Stop the business service in the old storage service system according to the stop data synchronization message.

[0135] Specifically, after receiving the stop data synchronization message, the old storage service system of each node disconnects the session connection with the ISCSI service client, wherein the business service refers to an interface service for receiving external I / O requests, such as the ISCSI service.

[0136] Specifically, by determining the amount of metadata in the message queue and comparing it with the first preset amount threshold, whether to send notification information is determined, and the timing of stopping the service is dynamically determined according to the actual remaining amount of metadata, so as to avoid stopping the service too early when there is still a lot of metadata, resulting in incomplete data synchronization.

[0137] Each node independently determines and sends notification information based on the amount of metadata in its own message queue. The upgrade service control node broadcasts the stop data synchronization message only after receiving notification information from all nodes. This distributed collaborative decision-making mechanism ensures the comprehensiveness and accuracy of the business stop decision. Only when all nodes meet the conditions for stopping the business, the entire cluster will stop the business uniformly, avoiding inconsistent cluster status caused by incomplete data synchronization of some nodes, which affects the processing of the I / O strong consistency model.

[0138] On the basis of the above-mentioned embodiment, after each node receives the stop data synchronization message and the old storage service system disconnects the session connection with the ISCSI service client, the operation of transferring the remaining metadata in the message queue to the shared storage pool and sending a response message to the new storage service system is still ongoing. In an optional example, when all the metadata in the message queue less than the first preset number threshold is transferred to the shared storage pool, the method further includes:

[0139] Detect the amount of metadata in the message queue, and when the amount of metadata in the message queue is less than or equal to a second preset amount threshold, send an end instruction to the new storage service system.

[0140] The end instruction is used to instruct the new storage service system to stop obtaining metadata from the shared storage pool, and the second preset quantity threshold is less than the first preset quantity threshold.

[0141] Specifically, when all the metadata in the message queue less than the first preset number threshold are transferred to the shared storage pool, the amount of metadata in the message queue is zero, the old storage service system sends an end instruction to the new storage service system, and the new storage service system obtains the remaining metadata from the shared storage pool and then stops obtaining metadata from the shared storage pool.

[0142] Specifically, the above mechanism can prevent the new storage service system from performing unnecessary data acquisition operations after the data has been completely synchronized, thereby wasting system resources.

[0143] Based on the above embodiments, the amount of metadata in the message queue will change dynamically as I / O requests are continuously generated. Depending on the real-time processing capability and network status of each node, the processing capability and network status of different nodes are different, which may affect the synchronization efficiency and system stability. Fixed thresholds may not be able to adapt to such dynamic changes.

[0144] In an optional embodiment, in order to improve the adaptability and efficiency of data synchronization, an adaptive threshold adjustment mechanism is proposed. The mechanism dynamically adjusts the first preset number threshold according to the real-time processing capability and network status of each node to ensure the efficiency and stability of the synchronization operation. The method specifically includes:

[0145] Step e1: The old storage service system monitors the processing capacity and network bandwidth of each node in real time.

[0146] Step e2: dynamically adjust the first preset quantity threshold according to the monitoring result.

[0147] For example, if a node has strong processing capabilities and sufficient network bandwidth, the threshold can be appropriately increased; if a node has weak processing capabilities or limited network bandwidth, the threshold can be appropriately lowered.

[0148] Step e3: when the amount of metadata in the message queue is less than the first preset amount threshold after dynamic adjustment, a notification message is sent to the upgrade service control node, indicating that the amount of remaining metadata in the message queue is less than or equal to the first preset amount threshold after dynamic adjustment.

[0149] The aforementioned embodiments are all executed by the old storage service system of each node. In this embodiment, a data synchronization method is provided. The method is executed by the new storage service system in the first node. The first node is any one of the multiple nodes and can be used for the above-mentioned mobile terminal, such as a mobile phone, a tablet computer, etc. Figure 4is a flow chart of a data synchronization method provided by an embodiment of the present invention, such as Figure 4 As shown, the process includes the following steps:

[0150] Step S401: Send a first data synchronization request to the old storage service system.

[0151] Specifically, after receiving the data synchronization instruction sent by the upgraded service control node, the new storage service system sends a first data synchronization request to the old storage service system, requesting the old storage service system to add metadata of the stored historical data to the pre-built shared storage pool.

[0152] Step S402: Receive a first response message fed back by the old storage service system.

[0153] The first response message is a response message sent by the old storage service system after adding the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request.

[0154] Step S403: after reading metadata of historical data from the shared storage pool according to the first response message, perform a metadata synchronization operation.

[0155] Step S404: After executing the metadata synchronization operation, a second data synchronization request is sent to the old storage service system.

[0156] The second data synchronization request is used to request the old storage service system to add the metadata in the message queue to the shared storage pool.

[0157] Step S405: Receive a second response message fed back by the old storage service system.

[0158] The second response message is a response message fed back by the old storage service system after adding the metadata in the message queue to the shared storage pool according to the second data synchronization request.

[0159] Step S406: According to the second response message, the metadata synchronization operation is performed again after reading the remaining metadata from the shared storage pool.

[0160] Specifically, the new storage service system sends a first data synchronization request to synchronize the metadata of historical data stored in the old storage service system. This ensures that the new storage service system can obtain the metadata information of the data accumulated in the old storage service system over a long period of time, providing a basis for subsequent data management and query operations. A second data synchronization request is sent to synchronize the newly added metadata in the message queue, ensuring that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner, ensuring the integrity of the metadata. The response message provides the new storage service system with clear data synchronization status information, and the new storage service system performs subsequent operations based on the response message, avoiding the new storage service system from constantly monitoring whether the old storage service system stores all metadata in the shared storage pool, resulting in a waste of resources.

[0161] On the basis of the above-mentioned embodiment, after the old storage service system receives the stop data synchronization message sent by the upgraded service control node, the old storage service system will disconnect the session connection with the business service client, but the action of transferring the metadata in the message queue to the shared storage pool is still in progress. When the amount of metadata in the message queue is reduced to a second preset threshold, for the new storage service system, the method further includes:

[0162] Step f1, receiving an end instruction sent by the old storage service system.

[0163] Specifically, when the amount of metadata in the message queue continues to decrease until it reaches a second preset threshold (for example, a value less than or equal to zero), the old storage service system will send an end instruction to the new storage service system to instruct the new storage service system to stop obtaining metadata from the shared storage pool after reading the remaining metadata from the shared storage pool.

[0164] The second preset threshold is a value less than or equal to zero.

[0165] Step f2: according to the end instruction, stop obtaining metadata from the shared storage pool.

[0166] Specifically, after the new storage service system reads the remaining metadata from the shared storage pool, it stops obtaining metadata from the shared storage pool. At this point, all metadata in the old storage service system has been transferred to the new storage service system.

[0167] Step f3: Start the business service and establish a session connection with the business service client.

[0168] Specifically, after all metadata in the old storage service system is transferred to the new storage service system, the new storage service system starts the business service, namely the ISCSI service. The cluster state of the new storage service system changes from the ready state to the prepared state, and processes I / O requests for clients through the ISCSI service.

[0169] After the upgrade service control node detects that the new cluster state has changed to the ready state, it broadcasts to stop the old storage service system, and each node deletes the related configuration files and releases the related resources. From then on, the upgrade is completed.

[0170] Specifically, a second preset threshold is set to ensure that the end instruction will not be sent until there is no metadata to be transferred in the old storage service system, thereby ensuring that the metadata is completely transferred.

[0171] After all metadata in the old storage service system is transferred to the new storage service system, an end instruction is issued. According to the end instruction, the new storage service system stops obtaining metadata from the shared storage pool, starts business services, and establishes a session connection with the new storage service system. This orderly switching process enables the business service client to complete the migration from the old storage service system to the new storage service system with almost no perception, ensuring business continuity.

[0172] As mentioned above, the overall process of online upgrade of distributed storage system is as follows: Figure 5 As shown, one node is selected from all nodes of the distributed storage system as the upgrade service control node, and the upgrade service is started to control and manage the entire cluster upgrade process.

[0173] First, the upgrade service control node broadcasts the upgrade environment detection, and each node detects the installation environment corresponding to the new storage service system according to the broadcast message. When all nodes meet the preset requirements for installing the new storage service system, a new cluster corresponding to the new storage service system is built. The specific construction method is as described above and will not be repeated here.

[0174] All nodes in the cluster perform data synchronization according to the broadcast message of the upgrade service control node, completing the data synchronization from the old cluster to the new cluster. The specific synchronization process of each node is as described above. After the data synchronization process is completed, the old cluster stops business services, cleans up related files and releases related resources, and the new cluster starts business services.

[0175] As mentioned above, the distributed storage system is composed of multiple nodes in a cluster working mode to provide unified services. All nodes in the cluster start and stop the data synchronization process according to the broadcast message of the upgrade service control node. The structural diagram of the data synchronization process of each node in the cluster is as follows: Figure 6 shown.

[0176] During the data synchronization process, each node has two storage service systems, including the old storage service system and the new storage service system. The new storage service systems of each node use a new cluster port that is different from the old cluster port to establish a communication connection link to ensure the normal execution of the multi-copy strong consistency model of the old cluster during data synchronization. Each node is connected to an ISCSI client. The client sends an I / O request to the node through the ISCSI service. After the node processes the request, it returns the processing result through the ISCSI service. After data synchronization stops, the old session between the old storage service system and the ISCSI client will be stopped, and a new session between the new storage service system and the ISCSI client will be connected.

[0177] Finally, a specific example will be used to illustrate the entire interactive process of the data synchronization method between the old storage service system and the new storage service system of any node in the cluster, including:

[0178] Step g1: After the cluster corresponding to the new storage service system is successfully built, the cluster status corresponding to the new storage service system is set to ready.

[0179] Step g2: After the upgrade service control node detects that the new cluster is in a ready state, it broadcasts a command carrying a data synchronization start message.

[0180] Step g3: the old storage service system creates a shared storage pool according to the instruction carrying the data synchronization start message.

[0181] Step g4: The new storage service system in each node sends a request to the old storage service system to create a communication connection link according to the data synchronization start message.

[0182] The communication connection link may be a Transmission Control Protocol (TCP for short).

[0183] Step g5: the old storage service system responds to the request to create a communication connection link sent by the new storage service system, and establishes a communication connection link between the old storage service system and the new storage service system.

[0184] Step g6: The new storage service system sends a first data synchronization request to the old storage service system using the communication connection link.

[0185] Step g7: After receiving the first data synchronization request, the old storage service system obtains the metadata of the historical data stored in the disk and adds it to the pre-built shared storage pool according to the pre-built format requirements. The metadata of the newly obtained data corresponding to the I / O request is added to the pre-built message queue, and the first response message is sent to the new storage service system.

[0186] Step g8: After the new storage service system receives the first response message fed back by the old storage service system, it reads the metadata corresponding to the historical data from the shared storage pool, and stores the metadata corresponding to the historical data in the new storage service system, and sends a second data synchronization request to the old storage service system, requesting to obtain the metadata in the message queue.

[0187] Step g9: the old storage service system determines the amount of metadata in the message queue according to the second data synchronization message, and determines whether the amount of metadata in the message queue is greater than a first preset threshold.

[0188] Step g901, when the amount of metadata in the message queue is greater than a first preset threshold (for example, 5 metadata), the metadata in the message queue is added to the shared storage pool, and a second response message is sent to the new storage service system.

[0189] Step g902: When the amount of metadata in the message queue is less than or equal to the first preset amount threshold (e.g., 5 metadata), the metadata in the message queue is added to the shared storage pool, and a second response message is sent to the new storage service system. A notification message is also sent to the upgrade service control node, indicating that the amount of remaining metadata in the message queue is less than or equal to the first preset amount threshold.

[0190] Step g10: The new storage service system receives the second response message fed back by the old storage service system, and reads the remaining metadata from the shared storage pool according to the second response message.

[0191] Step g11: After receiving notification information fed back by all nodes, the upgrade service control node broadcasts a message to stop data synchronization.

[0192] Step g12: According to the stop data synchronization message, the old storage service system disconnects the session connection with the ISCSI service client and stops processing I / O requests for the client.

[0193] Step g13, the old storage service system continues to add the metadata in the message queue to the shared storage pool until the amount of metadata in the message queue is lower than a second preset threshold (for example, the second preset threshold is a number less than or equal to zero), and sends an end instruction to the new storage service system.

[0194] Step g14, the new storage service system receives the end instruction sent by the old storage service system. According to the end instruction, the new storage service system reads the remaining metadata from the shared storage pool and stops obtaining metadata from the shared storage pool. At this point, all metadata in the old storage service system has been transferred to the new storage service system.

[0195] Step g15: The new storage service system establishes a session connection with the ISCSI service client. The cluster state of the new storage service system changes from the ready state to the prepared state, and processes I / O requests for the client through the ISCSI service.

[0196] In this embodiment, a data synchronization device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0197] This embodiment provides a data synchronization device, such as Figure 7 , including: a receiving module 701, a transfer module 702, and a sending module 703.

[0198] Receiving module 701, used to receive a first data synchronization request sent by the new storage service system;

[0199] The transfer module 702 is used to add the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and add the metadata of the data corresponding to the newly obtained I / O request to the pre-built message queue;

[0200] A sending module 703 is used to send a first response message to the new storage service system, where the first response message is used to instruct the new storage service system to perform a data synchronization operation on the new storage service system after reading metadata of historical data from the shared storage pool;

[0201] The receiving module 701 is further used to obtain a second data synchronization request sent by the new storage service system;

[0202] The transfer module 702 is further used to add the metadata in the message queue to the shared storage pool according to the second data synchronization request;

[0203] The sending module 703 is further used to send a second response message to the new storage service system, where the second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool and then perform the data synchronization operation on the new storage service system again.

[0204] In an optional example, the receiving module 701 is specifically configured to receive a data synchronization instruction issued by the upgrade service control node;

[0205] The creation module 704 is specifically configured to create a shared storage pool according to the data synchronization instruction.

[0206] In an optional example, the receiving module 701 is further configured to receive a communication connection link creation request sent by the new storage service system;

[0207] Processing module 705 is specifically used to create a communication connection link between the new storage service system and the communication connection link creation request, so that the new storage service system sends a data synchronization request through the communication connection link, wherein the data synchronization request includes a first data synchronization request and a second data synchronization request.

[0208] In an optional example, the processing module 705 is further used to determine the amount of metadata in the message queue;

[0209] The sending module 703 is further configured to send notification information to the upgrade service control node when the amount of metadata in the message queue is less than a first preset amount threshold, where the notification information is used to indicate that the amount of remaining metadata in the message queue is less than or equal to the first preset amount threshold;

[0210] The receiving module 701 is further used to receive a stop data synchronization message sent by the upgrade service control node. The stop data synchronization message is a stop data synchronization message broadcasted by the upgrade service control node to all nodes after receiving notification information fed back by all nodes respectively;

[0211] The processing module 705 is further used to stop the business service in the old storage service system according to the stop data synchronization message.

[0212] In an optional example, the sending module 703 is also used to detect the amount of metadata in the message queue, and when the amount of metadata in the message queue is less than or equal to a second preset number threshold, an end instruction is sent to the new storage service system, wherein the end instruction is used to instruct the new storage service system to stop obtaining metadata from the shared storage pool, and the second preset number threshold is less than the first preset number threshold.

[0213] In an optional example, the sending module 703 is further configured to send a first data synchronization request to the old storage service system;

[0214] The receiving module 701 is also used to receive a first response message fed back by the old storage service system, where the first response message is a response message sent by the old storage service system after adding the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request;

[0215] The reading module 706 is specifically configured to read the metadata of the historical data from the shared storage pool according to the first response message and then perform a metadata synchronization operation;

[0216] The sending module 703 is further used to send a second data synchronization request to the old storage service system after performing the metadata synchronization operation;

[0217] The receiving module 701 is further used to receive a second response message fed back by the old storage service system, where the second response message is a response message fed back by the old storage service system after adding the metadata in the message queue to the shared storage pool according to the second data synchronization request;

[0218] The reading module 706 is further configured to read the remaining metadata from the shared storage pool according to the second response message and then perform the metadata synchronization operation again.

[0219] In an optional example, the receiving module 701 is further configured to receive an end instruction sent by the old storage service system;

[0220] The processing module 705 is further used to stop obtaining metadata from the shared storage pool according to the end instruction;

[0221] Start the business service and establish a session connection with the business service client.

[0222] The data synchronization device in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0223] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0224] A data synchronization device provided by an embodiment of the present invention maintains the business service of the old storage service system at the front end, and the new storage service system at the back end sends a first data synchronization request. The old storage service system adds the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and then feeds back a first response message. It ensures that the new storage service system can obtain the metadata information of the data accumulated in the old system for a long time. The new storage service system sends a second data synchronization request, and the old storage service system adds the metadata in the message queue to the shared storage pool, and sends a second response message to the new storage service system. It ensures that the metadata of the data corresponding to the new I / O request generated during the data synchronization process can also be synchronized to the new storage service system in a timely manner. The two data synchronization requests and response messages ensure that the new storage service system can not only obtain the metadata of the historical data, but also synchronize the metadata of the newly added data in a timely manner. In addition, the data synchronization process does not interrupt the reception and processing of the I / O request, so that the front-end business interface of the data synchronization process is not perceived.

[0225] During the data synchronization process, the processing of the I / O strong consistency model continues to be handled by the entire old cluster. After the data synchronization stops, the new cluster takes over. Since the data synchronization process does not interrupt the reception and processing of I / O requests, there is no need to restart some services. This cleverly avoids considering the processing of the I / O strong consistency model, and enables seamless and rapid switching of business services from the old storage service system to the new storage service system.

[0226] An embodiment of the present invention further provides a computer device, Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0227] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit. The integrated circuit may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0228] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0229] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0230] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0231] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0232] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0233] The embodiment of the present invention also provides a computer-readable storage medium. The method provided in the above embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or is implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0234] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0235] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method is applied to a distributed storage system, the distributed storage system includes multiple nodes for performing upgrade services, each of the nodes includes an old storage service system and an upgraded new storage service system, the method is performed by the old storage service system in a first node, the first node is any one of the multiple nodes, the method includes: Receiving a first data synchronization request sent by the new storage service system; According to the first data synchronization request, metadata of the historical data stored in the old storage service system is added to a pre-built shared storage pool, and metadata of the data corresponding to the newly obtained I / O request is added to a pre-built message queue; Sending a first response message to the new storage service system, where the first response message is used to instruct the new storage service system to perform a data synchronization operation on the new storage service system after reading the metadata of the historical data from the shared storage pool; Obtaining a second data synchronization request sent by the new storage service system; According to the second data synchronization request, the metadata in the message queue is added to the shared storage pool, and a second response message is sent to the new storage service system. The second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool and then perform data synchronization operations on the new storage service system again.

2. The method according to claim 1, characterized in that The distributed storage system includes a plurality of nodes for executing an upgrade service, wherein the plurality of nodes include an upgrade service control node. Before receiving the first data synchronization request sent by the new storage service system, the method further includes: Receiving a data synchronization instruction issued by the upgrade service control node; The shared storage pool is created according to the data synchronization instruction.

3. The method according to claim 1, characterized in that Before receiving the first data synchronization request sent by the new storage service system, the method further includes: Receiving a communication connection link creation request sent by the new storage service system; According to the communication connection link creation request, a communication connection link is created between the new storage service system and the new storage service system so that the new storage service system sends a data synchronization request through the communication connection link, wherein the data synchronization request includes the first data synchronization request and the second data synchronization request.

4. The method according to claim 2, characterized in that: Before adding the metadata in the message queue to the shared storage pool according to the second data synchronization request, the method further includes: determining the amount of metadata in the message queue; When the amount of metadata in the message queue is less than a first preset amount threshold, sending notification information to the upgrade service control node, where the notification information is used to indicate that the amount of remaining metadata in the message queue is less than or equal to the first preset amount threshold; receiving a stop data synchronization message sent by the upgrade service control node, wherein the stop data synchronization message is broadcasted by the upgrade service control node to all nodes after receiving the notification information fed back by all nodes respectively; According to the stop data synchronization message, the business service in the old storage service system is stopped.

5. The method according to claim 4, characterized in that After stopping the business service in the old storage service system according to the stop data synchronization message, the method further includes: Detect the amount of metadata in the message queue, and when the amount of metadata in the message queue is less than or equal to a second preset number threshold, send an end instruction to the new storage service system, wherein the end instruction is used to instruct the new storage service system to stop obtaining metadata from the shared storage pool, and the second preset number threshold is less than the first preset number threshold.

6. A data synchronization method, characterized in that: The method is applied to a distributed storage system, the distributed storage system includes multiple nodes for performing upgrade services, each of the nodes includes an old storage service system and an upgraded new storage service system, the method is performed by the new storage service system in a first node, the first node is any one of the multiple nodes, the method includes: Sending a first data synchronization request to the old storage service system; Receive a first response message fed back by the old storage service system, where the first response message is a response message sent by the old storage service system after adding metadata of historical data stored in the old storage service system to a pre-built shared storage pool according to the first data synchronization request; After reading the metadata of the historical data from the shared storage pool according to the first response message, a metadata synchronization operation is performed; After executing the metadata synchronization operation, sending a second data synchronization request to the old storage service system; Receive a second response message fed back by the old storage service system, where the second response message is a response message fed back by the old storage service system after adding the metadata in the message queue to the shared storage pool according to the second data synchronization request; According to the second response message, the metadata synchronization operation is performed again after reading the remaining metadata from the shared storage pool.

7. The method according to claim 6, characterized in that The method further comprises: Receiving an end instruction sent by the old storage service system; According to the end instruction, stop acquiring metadata from the shared storage pool; Start the business service and establish a session connection with the business service client.

8. A data synchronization device, characterized in that: The device comprises: A receiving module, used for receiving a first data synchronization request sent by the new storage service system; A transfer module, used to add the metadata of the historical data stored in the old storage service system to the pre-built shared storage pool according to the first data synchronization request, and to add the metadata of the data corresponding to the newly obtained I / O request to the pre-built message queue; A sending module, used for sending a first response message to the new storage service system, wherein the first response message is used for instructing the new storage service system to perform a data synchronization operation on the new storage service system after reading the metadata of the historical data from the shared storage pool; The receiving module is further used to obtain a second data synchronization request sent by the new storage service system; The transfer module is further configured to add the metadata in the message queue to the shared storage pool according to the second data synchronization request; The sending module is further used to send a second response message to the new storage service system, wherein the second response message is used to instruct the new storage service system to read the remaining metadata from the shared storage pool and then perform a data synchronization operation on the new storage service system again.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data synchronization method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data synchronization method according to any one of claims 1 to 7.

Images