Method, system, apparatus, device and medium for data transmission among multiple sites

Abstract

This disclosure provides a method, system, apparatus, device, and medium for data transmission between multiple sites. The method includes: responding to a data copy command, controlling all storage nodes in the original data pool of a first site to perform a data copy task, copying the data stripe to be migrated to a free data pool of a sub-site located in the first site; when the data stripe to be migrated is completely copied to the free data pool, controlling the free data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and storing the migration verification fragment in the free data pool; powering down the storage devices in the free data pool to transport the free data pool to a second site via physical transport, enabling the second site to read the data stripe to be migrated and the migration verification fragment in the free data pool and verify the data stripe to be migrated. This method employs automation, reducing the tediousness and error risk of manual operation, and ensuring the integrity and security of data during transmission.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, system, apparatus, device and medium for data transmission between multiple sites. Background Technology

[0002] Large enterprises or organizations often need to establish branches in different regions due to business needs. These branches frequently require the transmission of large amounts of business data to data processing centers. For example, observation stations located far from data processing centers collect large amounts of raw data, which needs to be transmitted to the data processing center in a timely manner for further analysis and processing. As the accuracy and resolution of data acquisition continue to improve, the amount of data that needs to be transmitted also increases significantly.

[0003] To address this data transfer requirement, the technology employs a truck-transported storage device method. This involves copying data from branch offices to temporary storage devices, transporting these devices to the data center by truck, and finally copying the data from the temporary storage devices to the central storage system. This process requires manual operation of the data migration software and manual verification of the data's correctness and integrity, increasing the complexity and risk of errors. Summary of the Invention

[0004] This application provides a method, system, apparatus, equipment, and medium for data transmission between multiple sites. It employs automated means to complete data copying and generate verification fragments at the first site, and then automatically reads and verifies these copied data at the second site. This simplifies the steps of cross-site data transmission, reduces the tediousness and error risk of manual operation, and ensures the integrity and security of data during transmission.

[0005] A first aspect of this disclosure provides a multi-site data transmission method applied to a first site, comprising: responding to a data copy command, controlling all storage nodes in the original data pool of the first site to perform a data copy task, copying the data stripe to be migrated to a free data pool, the free data pool being a sub-site within the first site; when the data stripe to be migrated is completely copied to the free data pool, controlling the free data pool to generate a migration check fragment corresponding to the data stripe to be migrated, and storing the migration check fragment in the free data pool; powering down the storage devices in the free data pool to transport the free data pool to a second site via physical transport, enabling the second site to read the data stripe to be migrated and the migration check fragment in the free data pool and to verify the data stripe to be migrated.

[0006] In some embodiments of this disclosure, when the data stripe to be migrated is completely copied to the detached data pool, controlling the detached data pool to generate migration check fragments corresponding to the data stripe to be migrated and storing the migration check fragments in the detached data pool includes: controlling the detached data pool to generate migration check fragments corresponding to the data stripe to be migrated according to the number of check fragments indicated by the erasure coding algorithm, using the check algorithm, and storing the data stripe to be migrated and the migration check fragments in different storage devices in the detached data pool respectively; controlling the detached data pool to transmit the migration check fragments to the second site via the network so that the second site stores the migration check fragments.

[0007] In some embodiments of this disclosure, when the data stripe to be migrated is completely copied to the detached data pool, the detached data pool is controlled to generate a migration check shard corresponding to the data stripe to be migrated, and the migration check shard is stored in the detached data pool. Then, the method includes: updating the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronizing the updated metadata to the second metadata pool of the second site. The updated metadata indicates the storage device location of the data stripe to be migrated and the migration check shard in the detached data pool.

[0008] In some embodiments of this disclosure, in response to a data copy instruction, controlling all storage nodes in the original data pool of the first site to perform a data copy task and copy the data stripe to be migrated to the free data pool includes: obtaining the historical performance value of the data copy task within a preset time period, predicting the future performance value of the data copy task within the preset time period based on the historical performance value, and adjusting the priority of the original data pool to perform the data copy task according to the future performance value.

[0009] A second aspect of this disclosure provides a method for data transmission between multiple sites, applied to a second site, comprising: responding to a storage device of a detached data pool accessing a sub-site of the second site; controlling the target data pool of the second site to read the data stripe to be migrated and the migration verification fragment in the detached data pool; and sequentially using the migration verification fragment to verify the data stripe to be migrated; the detached data pool is transported from the first site to the second site; the data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the detached data pool in the first site; if all the data stripes to be migrated are successfully verified, the detached state of the detached data pool is adjusted to a normal state.

[0010] In some embodiments of this disclosure, in response to the storage device of the detached data pool accessing a sub-site of the second site, the target data pool of the second site is controlled to read the data stripe to be migrated and the migration check fragment in the detached data pool, and the migration check fragment is used to check the data stripe to be migrated in sequence. This includes: in response to the storage device of the detached data pool accessing a sub-site of the second site, updating the routing table of the detached data pool; when the routing table of the detached data pool is updated, reading the data stripe to be migrated and the migration check fragment in the detached data pool according to the metadata in the second metadata pool of the second site, the metadata indicating the location of the data stripe to be migrated and the migration check fragment in the storage device of the detached data pool; and checking the data stripe to be migrated in sequence according to the read migration check fragment and the migration check fragment stored in the second site, wherein the migration check fragment stored in the second site is obtained through network transmission when the detached data pool is located at the first site.

[0011] In some embodiments of this disclosure, the migration check shards to be migrated are checked sequentially based on the read migration check shards and the migration check shards stored in the second site. This includes: performing a first check on the first data stripe to be migrated based on the read migration check shards; if the first check fails, then performing a second check on the data stripe to be migrated using the migration check shards stored in the second site; if the second check succeeds or the first check succeeds, then the first data stripe to be migrated is determined to have been successfully checked; if the second check fails, then the first data stripe to be migrated fails, and data recovery is performed on the first data stripe to be migrated using the read migration check shards and the migration check shards stored in the second site.

[0012] A third aspect of this disclosure provides a multi-site data transmission system, comprising: a first site management system located at a first site and a second site management system located at a second site; the first site management system is configured to, in response to a data copy command, control all storage nodes in the original data pool of the first site to perform a data copy task, copying the data stripe to be migrated to the detached data pool of a sub-site located in the first site; when the data stripe to be migrated is completely copied to the detached data pool, control the detached data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and store the migration verification fragment in the detached data pool; and power down the storage devices in the detached data pool to transport the detached data pool to the second site via physical transport; the second site management system is configured to, in response to the storage devices of the detached data pool connecting to the sub-site of the second site, control the target data pool of the second site to read the data stripe to be migrated and the migration verification fragment in the detached data pool, and sequentially use the migration verification fragment to verify the data stripe to be migrated; if all the data stripes to be migrated are successfully verified, the detached state of the detached data pool is adjusted to a normal state.

[0013] A fourth aspect of this disclosure provides a multi-site data transmission apparatus applied to a first site, comprising: a copy unit, configured to, in response to a data copy command, control all storage nodes in the original data pool of the first site to perform a data copy task, copying the data stripe to be migrated to a free data pool, the free data pool being a sub-site within the first site; a verification fragment generation unit, configured to, when the data stripe to be migrated is completely copied to the free data pool, control the free data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and store the migration verification fragment in the free data pool; and a migration unit, configured to power down the storage devices in the free data pool to transport the free data pool to a second site via physical transport, enabling the second site to read the data stripe to be migrated and the migration verification fragment in the free data pool and to verify the data stripe to be migrated.

[0014] A fifth aspect embodiment of this disclosure provides a multi-site data transmission apparatus applied to a second site, comprising: a data verification unit, configured to, in response to a storage device of a detached data pool accessing a sub-site of the second site, control the target data pool of the second site to read the data stripe to be migrated and the migration verification fragment in the detached data pool, and sequentially verify the data stripe to be migrated using the migration verification fragment, wherein the detached data pool is transported from the first site to the second site, and the data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the detached data pool in the first site; and a state adjustment unit, configured to, if all the data stripes to be migrated are successfully verified, adjust the detached state of the detached data pool to a normal state.

[0015] A sixth aspect of this disclosure provides an electronic device including: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method described in the first aspect of this disclosure.

[0016] A seventh aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0017] In summary, the multi-site data transmission method provided in this disclosure includes: responding to a data copy command, controlling all storage nodes in the original data pool of a first site to perform a data copy task, copying the data stripe to be migrated to a free data pool, the free data pool being a sub-site within the first site; when the data stripe to be migrated is completely copied to the free data pool, controlling the free data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and storing the migration verification fragment in the free data pool; powering down the storage devices in the free data pool to transport the free data pool to a second site via physical transport, enabling the second site to read the data stripe to be migrated and the migration verification fragment in the free data pool and verify the data stripe to be migrated. This method employs automation, copying the data stripe to the free data pool and generating a verification fragment at the first site, then physically transporting it to the second site. The second site automatically reads the data stripe and verification fragment for verification, simplifying the steps of cross-site data transmission, reducing the tediousness and error risk of manual operation, and ensuring the integrity and security of data during transmission.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0020] Figure 1 This is a flowchart illustrating a method for data transmission between multiple sites provided in an embodiment of this disclosure.

[0021] Figure 2 This is a flowchart illustrating another method for data transmission between multiple sites provided in an embodiment of this disclosure;

[0022] Figure 3 This is a flowchart illustrating another method for data transmission between multiple sites provided in an embodiment of this disclosure;

[0023] Figure 4 This is a schematic diagram of the framework of a multi-site data transmission system provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of a specific multi-site data transmission system proposed in this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of a multi-site data transmission device provided in an embodiment of the present disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a multi-site data transmission device provided in an embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] Large enterprises or organizations often establish branch offices in different regions, requiring frequent transmission of large amounts of business data between these branch offices and data processing centers. For example, observation stations far from data processing centers collect a large amount of raw data, which needs to be transmitted to the data processing center for further processing. However, as the accuracy and resolution of data acquisition continue to improve, the amount of data that needs to be transmitted also increases dramatically.

[0030] While leased network lines offer relatively stable network connectivity, their bandwidth is limited. When large amounts of data need to be transmitted, bandwidth can become a bottleneck for transmission speed, resulting in long transmission times. For example, even with 200MB of bandwidth, transmitting 1PB of data can take up to two months, which is not only costly and time-consuming but also leads to a large backlog of raw data at the data collection sites.

[0031] To avoid network bandwidth bottlenecks, related technologies employ the method of transporting storage devices by truck. For example, a 20-ton truck can carry 40,000 hard drives, with a total capacity of up to 600,000 TB, thus transferring massive amounts of data at once. This method has significant advantages in handling large-scale data transfer tasks.

[0032] However, when transporting storage devices by truck, data needs to be copied from branch offices to mobile temporary storage devices, then transported by truck to the data center, and finally copied to the central storage system. This process involves data consistency issues across multiple storage systems, and data migration management and control between storage systems are relatively difficult. It requires not only manual operation of data migration software but also manual verification of data correctness and integrity, increasing operational complexity and the risk of errors.

[0033] In order to solve the technical problems existing in the related technologies, the embodiments of this disclosure provide a method for data transmission between multiple sites.

[0034] The embodiments of this disclosure will now be described in detail.

[0035] like Figure 1 As shown, embodiments of this disclosure provide a method for data transmission between multiple sites, applied to a first site, comprising the following steps:

[0036] Step 101: In response to the data copy command, control all storage nodes in the original data pool of the first site to perform data copy tasks, copy the data stripe to be migrated to the free data pool, which is located in a sub-site of the first site.

[0037] In some embodiments, users can trigger the generation of data copy instructions through management software operations within the first site management system, so that the first site management system can perform data migration operations based on the data copy instructions, that is, control all storage nodes in the original data pool of the first site to concurrently execute the data copy tasks corresponding to the data copy instructions, and copy the data strips to be migrated to the free data pool.

[0038] The raw data pool consists of multiple storage nodes in a distributed storage system. Its main function is to store initial, unprocessed data. To improve the efficiency and reliability of data access, this data is distributed across the storage nodes in a striped manner.

[0039] The detached data pool is pre-built and includes multiple storage devices. The main function of these storage devices is to store data stripes that have been copied out and are ready for migration.

[0040] When performing stripe copy operations on the data to be migrated, it is important to note that the storage nodes of the original data pool and the detached data pool are located within the same site (i.e., the same data center). This means that the detached data pool and the original data pool are interconnected via a local network, enabling efficient and rapid data transfer. This physical proximity reduces data transmission latency and bandwidth consumption.

[0041] The core step of the copy operation is to copy the data stripes to be migrated (i.e., the data fragments distributed across various storage nodes in the original data pool) from the original data pool to the storage device of the detached data pool. This process ensures data integrity and consistency, while also facilitating subsequent data processing or export operations.

[0042] In the process of copying data stripes to be migrated to the detached data pool by performing data copying tasks on all storage nodes of the distributed storage system in the original data pool, this disclosure also needs to adjust the priority of the data copying tasks performed in the original data pool in real time based on the historical performance values ​​of the data copying tasks.

[0043] Step 102: When the data stripe to be migrated is completely copied to the detached data pool, control the detached data pool to generate the migration check shard corresponding to the data stripe to be migrated, and store the migration check shard in the detached data pool.

[0044] In some embodiments, after the data stripes to be migrated are completely copied from the original data pool to the detached data pool, the detached data pool is further controlled to generate migration verification shards corresponding to these data stripes. Subsequently, these migration verification shards are also securely stored in the detached data pool to ensure that the integrity and accuracy of the data are verified during the migration process.

[0045] Step 103: Power off the storage devices in the detached data pool to transport the detached data pool to the second site via physical transport, so that the second site can read the data stripe to be migrated and the migration check fragment in the detached data pool and verify the data stripe to be migrated.

[0046] In some embodiments, to prepare for the physical transportation process and ensure that the storage devices are in a safe, low-power state during transportation, this disclosure performs a power-down (i.e., power-off) operation on the storage devices in the detached data pool after the data stripe to be migrated and its corresponding migration check fragment have been stored in the detached data pool. Subsequently, the detached data pool is transported to a second site by physical means (such as trucks, airplanes, etc.).

[0047] Upon arrival at the second site, the storage devices in the detached data pool will be reconnected, and the data stripes to be migrated and migration verification shards will be read. Using the migration verification shards, the second site will perform rigorous verification on each data stripe to be migrated to ensure that no data corruption or loss occurs during the data migration process, thus maintaining data integrity and reliability.

[0048] In summary, the multi-site data transmission method proposed in this disclosure includes: responding to a data copy command, controlling all storage nodes in the original data pool of the first site to perform a data copy task, copying the data stripe to be migrated to a free data pool, the free data pool being a sub-site within the first site; when the data stripe to be migrated is completely copied to the free data pool, controlling the free data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and storing the migration verification fragment in the free data pool; powering down the storage devices in the free data pool to transport the free data pool to the second site via physical transport, enabling the second site to read the data stripe to be migrated and the migration verification fragment in the free data pool and verify the data stripe to be migrated. This method employs automation, copying the data stripe to the free data pool and generating a verification fragment at the first site, then physically transporting it to the second site. The second site automatically reads the data stripe and the verification fragment for verification, simplifying the steps of cross-site data transmission, reducing the tediousness and error risk of manual operation, and ensuring the integrity and security of data during transmission.

[0049] Figure 2 A flowchart of a multi-site data transmission method proposed in this disclosure is further shown. Based on... Figure 1 The illustrated embodiments further explain, Figure 2 This may include the following steps.

[0050] Step 201: In response to the data copy command, control all storage nodes in the original data pool of the first site to perform data copy tasks, copy the data stripe to be migrated to the free data pool, which is located in a sub-site of the first site.

[0051] In some implementations, data copying tasks are performed on each data stripe of the file or object data to be transferred. These tasks are distributed to all storage nodes of the distributed storage system where the original data resides and executed concurrently. The original data is evenly distributed in the form of data stripes in the original data pool, so data copying also adopts a data stripe-level data copying method, copying each data stripe of the original data to the storage device detached from the data pool.

[0052] Since data migration is typically a time-consuming process, potentially taking days or weeks, excessively fast migration speeds or high bandwidth usage can negatively impact system performance. To address this, the data copy task scheduling is dynamically adjusted based on the read / write I / O activity of the first site in the original data pool. Specifically, this disclosure obtains historical performance values ​​(i.e., historical I / O performance values) of data copy tasks within a preset time period (e.g., hours). Based on these historical performance values, it predicts the future performance values ​​of data copy tasks within the preset time period (i.e., the activity level in the next 24 hours), and adjusts the priority of executing data copy tasks in the original data pool according to these future performance values.

[0053] To achieve this dynamic adjustment, this disclosure specifically uses a time-period request prediction formula to calculate the request load value for the next time period. This value is calculated based on the request load value of the previous time period, the current request load statistics, and the attenuation coefficient. Then, the priority of the data copy task is determined according to the prediction results and the task scheduling priority formula.

[0054] The time period request prediction formula is:

[0055] Among them, Q t+1 The request load value for the next preset time period; Q t-1 This represents the request load value for the previous preset time period; Q t This represents the current request load statistics; λ is the attenuation coefficient.

[0056] The task scheduling priority formula is:

[0057] Where P0 is the task scheduling priority, Q t+1 Q represents the request load value for the next preset time period. max This is the preset maximum request load value.

[0058] In addition, to ensure that the performance within the original data pool site is not affected by data copying and verification, I / O requests for data copying, I / O requests for fault repair, and I / O requests for data read and write are scheduled and controlled in order of priority from low to high.

[0059] Step 202: When the data stripe to be migrated is completely copied to the detached data pool, the detached data pool is controlled to generate migration check fragments corresponding to the data stripe to be migrated according to the number of check fragments indicated by the erasure coding algorithm. The data stripe to be migrated and the migration check fragments are stored in different storage devices in the detached data pool respectively.

[0060] In some embodiments, to improve data reliability and fault tolerance, and to enable data recovery via other storage devices and parity shards even if some storage devices fail, this disclosure allows the detached data pool to determine the required number of parity shards according to the erasure coding algorithm. Using the parity algorithm, corresponding migration parity shards are generated for each data stripe to be migrated. The data stripes to be migrated and the generated migration parity shards are then stored separately on different storage devices within the detached data pool.

[0061] Furthermore, to provide greater data redundancy and disaster recovery capabilities (because if the first site fails, the second site still has the verification shards needed to recover the data), this disclosure allows for further control over the detached data pool to transmit the migrated verification shards over the network to the second site, enabling the second site to store the migrated verification shards. The second site can be a data center geographically distant from the first site.

[0062] In addition, to improve disaster recovery capabilities, the original data pool disclosed herein can also employ two data redundancy methods: replication technology or erasure coding algorithm.

[0063] Specifically, this disclosure can control the original data pool, divide the original data into multiple original data stripes, and copy each original data stripe to the storage node corresponding to the number of replicas indicated by the replication technology. Alternatively, it can control the original data pool, divide the original data into multiple original data stripes according to the number of data shards and the number of parity shards indicated by the erasure coding algorithm, and generate original parity shards corresponding to multiple original data stripes through the parity algorithm and the number of parity shards, so as to store the multiple original data stripes and original parity shards in different storage nodes. The multiple original data stripes include the data stripe to be migrated.

[0064] Step 203: Update the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronize the updated metadata to the second metadata pool of the second site. The updated metadata indicates the storage device location of the data stripe to be migrated and the migration check shard in the detached data pool.

[0065] In some embodiments, after the data stripe to be migrated is successfully copied to the detached data pool and the corresponding migration check shard is generated and stored, the relevant metadata in the first metadata pool of the first site needs to be updated. The updated metadata should contain accurate information about the storage device location of the data stripe to be migrated and the migration check shard in the detached data pool.

[0066] To ensure data consistency and disaster recovery capabilities, updated metadata needs to be synchronized to the second metadata pool at the second site. This synchronization process can be achieved via network transmission, ensuring that the metadata pool at the second site remains synchronized with the metadata pool at the first site.

[0067] Step 204: Power off the storage devices in the detached data pool to transport the detached data pool to the second site via physical transport, so that the second site can read the data stripe to be migrated and the migration check fragment in the detached data pool and verify the data stripe to be migrated.

[0068] In some embodiments, step 204 is the same as the principle described in step 103 above, and will not be repeated here.

[0069] In summary, the multi-site data transmission method proposed in this disclosure achieves efficient data migration and redundant storage, while dynamically adjusting the migration speed to avoid negative impacts on system performance and ensuring data consistency and disaster recovery capabilities.

[0070] like Figure 3 As shown, embodiments of this disclosure provide a method for data transmission between multiple sites, applied to a second site, comprising the following steps:

[0071] Step 301: In response to the storage device of the detached data pool being connected to the sub-site of the second site, the target data pool of the second site is controlled to read the data stripe to be migrated and the migration verification fragment in the detached data pool, and the migration verification fragment is used to verify the data stripe to be migrated in sequence. The detached data pool is transported from the first site to the second site, and the data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the detached data pool in the first site.

[0072] In some embodiments, after the storage device of the detached data pool is connected to a sub-site of the second site, the target data pool of the second site begins to read the data stripes to be migrated and the migration check fragments transported from the detached data pool of the first site. Subsequently, the target data pool uses the migration check fragments to perform verification on the data stripes to be migrated one by one. This verification mechanism is based on erasure coding algorithms, which can detect and correct errors that may occur during data transmission or storage, thereby ensuring data integrity and accuracy.

[0073] Specifically, this disclosure can respond to the access of the storage device of the detached data pool to the sub-site of the second site, update the routing table of the detached data pool, so that the accessing client can find the data stripe to be migrated in the detached data pool; when the routing table of the detached data pool is updated, the data stripe to be migrated and the migration verification fragment of the detached data pool are read according to the metadata in the second metadata pool of the second site, and the metadata indicates the location of the data stripe to be migrated and the migration verification fragment in the storage device of the detached data pool; according to the read migration verification fragment and the migration verification fragment stored in the second site, the data stripe to be migrated is verified in sequence, and the migration verification fragment stored in the second site is obtained through network transmission when the detached data pool is located at the first site.

[0074] Specifically, based on the read migration verification shards and the migration verification shards stored in the second site, the data stripes to be migrated are verified sequentially. This includes: performing a first verification on the first data stripe to be migrated based on the read migration verification shards; if the first verification fails, then performing a second verification on the data stripe to be migrated using the migration verification shards stored in the second site; if the second verification succeeds or the first verification succeeds, then the first data stripe to be migrated is determined to have been verified successfully; if the second verification fails, then the first data stripe to be migrated fails to be verified, and data recovery is performed on the first data stripe to be migrated using the read migration verification shards and the migration verification shards stored in the second site.

[0075] Step 302: If all the data stripes to be migrated are successfully verified, the detached state of the detached data pool is adjusted to the normal state.

[0076] In some embodiments, if all the data stripes to be migrated successfully pass the verification, it means that the data migration process is successful and the data has not been damaged during transmission. At this point, the status field in the metadata of the second metadata pool corresponding to the data stripe to be migrated can be changed from a copied state to an independent state. Then, the status of the detached data pool can be adjusted from "detached" to "normal". This status adjustment signifies that the data has been successfully migrated and stored in the target data pool of the second site, and has passed the complete verification process; it can now be considered normal and usable data.

[0077] In summary, this disclosure, by connecting a storage device in response to a detached data pool to a sub-site of a second site, controls the target data pool of the second site to read the data stripes to be migrated and the migration check fragments from the detached data pool, and sequentially uses the migration check fragments to verify the data stripes to be migrated. The detached data pool is transported from the first site to the second site, and the data stripes to be migrated and the migration check fragments are obtained from the original data pool and the detached data pool in the first site. If all the data stripes to be migrated are successfully verified, the detached state of the detached data pool is adjusted to a normal state. This achieves automation by copying the data stripes to the detached data pool and generating check fragments at the first site, and then physically transporting them to the second site. The second site automatically reads the data stripes and check fragments for verification, simplifying the steps of cross-site data transmission, reducing the tediousness and error risk of manual operation, and ensuring the integrity and security of data during transmission.

[0078] Figure 4 A schematic diagram of the framework of a multi-site data transmission system proposed in this disclosure is further shown, which can be applied to... Figures 1 to 3 The method shown.

[0079] In some embodiments, a first site management system 1 is located at a first site and a second site management system 2 is located at a second site.

[0080] The first site includes the raw data pool, the free data pool, and the first metadata pool, while the second site includes the target data pool and the second metadata pool.

[0081] Specifically, for ease of understanding, such as Figure 5 The diagram shown illustrates a specific multi-site data transmission system provided in this disclosure. (Refer to...) Figure 5 , Figure 5 The original site in this disclosure is the first site, the application system in the original site is the first site management system, the target site is the second site in this disclosure, the application system in the target site is the second site management system, the metadata pool in the original site is the first metadata pool, and the metadata pool in the target site is the second metadata pool. This disclosure can determine the data to be migrated from files or objects.

[0082] The first site management system 1, in response to data copy commands, controls all storage nodes in the original data pool of the first site to perform data copy tasks, copying the data stripe to be migrated to the detached data pool of the sub-site located in the first site. When the data stripe to be migrated is completely copied to the detached data pool, the system controls the detached data pool to generate a migration verification fragment corresponding to the data stripe to be migrated, and stores the migration verification fragment in the detached data pool. The system then powers down the storage devices in the detached data pool to transport the detached data pool to the second site via physical transport. The data to be migrated can specifically be data in a file or object.

[0083] The second site management system 2 is used to respond to the access of the storage device of the detached data pool to the sub-site of the second site, control the target data pool of the second site to read the data strips to be migrated and the migration verification fragments in the detached data pool, and use the migration verification fragments to verify the data strips to be migrated in sequence; if all the data strips to be migrated are successfully verified, the detached state of the detached data pool is adjusted to the normal state.

[0084] In some embodiments, the first site management system 1 is used to control the detached data pool to generate migration check fragments corresponding to the data stripe to be migrated according to the number of check fragments indicated by the erasure coding algorithm, and to store the data stripe to be migrated and the migration check fragments in different storage devices in the detached data pool respectively; and to control the detached data pool to transmit the migration check fragments to the second site via the network, so that the second site stores the migration check fragments (corresponding to the data stripe to be migrated). Figure 4 (Target data pool checksum).

[0085] The first site management system is used to control the free data pool to generate migration check shards corresponding to the data stripe to be migrated when the data stripe to be migrated is completely copied to the free data pool, and to store the migration check shards in the free data pool. Then, it updates the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronizes the updated metadata to the second metadata pool of the second site. The updated metadata indicates the storage device location of the data stripe to be migrated and the migration check shards in the free data pool.

[0086] In this process, when the first site management system controls all storage nodes in the original data pool to perform data copying tasks and copy the data stripes to be migrated to the free data pool, the first site management system 1 is also used to obtain the historical performance value of the data copying tasks within a preset time period, and based on the historical performance value, predict the future performance value of the data copying tasks within the preset time period, so as to adjust the priority of the original data pool to perform data copying tasks according to the future performance value.

[0087] In some embodiments, the second site management system is configured to update the routing table of the detached data pool in response to the access of the storage device of the detached data pool to the sub-site of the second site; when the routing table of the detached data pool is updated, the system reads the data stripe to be migrated and the migration verification fragment of the detached data pool according to the metadata in the second metadata pool of the second site, the metadata indicating the location of the data stripe to be migrated and the migration verification fragment in the storage device of the detached data pool; and verify the data stripe to be migrated sequentially according to the read migration verification fragment and the migration verification fragment stored in the second site, the migration verification fragment stored in the second site being obtained through network transmission when the detached data pool is located at the first site.

[0088] The second site management system is used to perform a first verification on the first data stripe to be migrated based on the read migration verification shards and the migration verification shards stored in the second site when verifying the data stripe to be migrated in sequence. If the first verification fails, the migration verification shards stored in the second site are used to perform a second verification on the data stripe to be migrated. If the second verification succeeds or the first verification succeeds, the first data stripe to be migrated is determined to have been verified successfully. If the second verification fails, the first data stripe to be migrated fails to be verified, and the read migration verification shards and the migration verification shards stored in the second site are used to recover the data from the first data stripe to be migrated.

[0089] To implement the inter-site data transmission method provided in this disclosure, this disclosure also provides an inter-site data transmission apparatus, applied to a first site, such as... Figure 6 As shown, the multi-site data transmission device 600 includes:

[0090] The copy unit 610 is used to respond to the data copy command and control all storage nodes in the original data pool of the first site to perform data copy tasks, copying the data stripe to be migrated to the free data pool, which is located in a sub-site of the first site;

[0091] The verification shard generation unit 620 is used to control the free data pool to generate migration verification shards corresponding to the data stripe to be migrated when the data stripe to be migrated is completely copied to the free data pool, and to store the migration verification shards in the free data pool.

[0092] The migration unit 630 is used to power down the storage devices in the detached data pool so as to transport the detached data pool to the second site through physical transportation, so that the second site can read the data stripe to be migrated and the migration check fragment in the detached data pool and verify the data stripe to be migrated.

[0093] In some embodiments, the verification fragment generation unit 620 is configured to: control the detached data pool to generate migration verification fragments corresponding to the data stripe to be migrated according to the number of verification fragments indicated by the erasure coding algorithm, and store the data stripe to be migrated and the migration verification fragments in different storage devices in the detached data pool respectively; control the detached data pool to transmit the migration verification fragments to the second site via the network, so that the second site stores the migration verification fragments.

[0094] In some embodiments of this disclosure, the apparatus further includes an update unit, configured to, when the data stripe to be migrated is completely copied to the detached data pool, control the detached data pool to generate a migration check shard corresponding to the data stripe to be migrated, and store the migration check shard in the detached data pool. Then, update the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronize the updated metadata to the second metadata pool of the second site. The updated metadata indicates the storage device location of the data stripe to be migrated and the migration check shard in the detached data pool.

[0095] In some embodiments of this disclosure, the copy unit 610 is configured to: obtain historical performance values ​​of data copy tasks within a preset time period, predict future performance values ​​of data copy tasks within the preset time period based on the historical performance values, and adjust the priority of the original data pool in executing data copy tasks according to the future performance values.

[0096] To implement the inter-site data transmission method provided in this disclosure, this disclosure also provides an inter-site data transmission apparatus, applied to a second site, such as... Figure 7 As shown, the multi-site data transmission device 700 includes:

[0097] The data verification unit 710 is used to respond to the access of the storage device of the free data pool to the sub-site of the second site, control the target data pool of the second site to read the data stripe to be migrated and the migration verification fragment in the free data pool, and use the migration verification fragment to verify the data stripe to be migrated in sequence. The free data pool is transported from the first site to the second site, and the data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the free data pool in the first site.

[0098] The status adjustment unit 720 is used to adjust the detached status of the detached data pool to the normal status if all the data stripes to be migrated are successfully verified.

[0099] In some embodiments, the data verification unit 710 is configured to: update the routing table of the detached data pool in response to the storage device of the detached data pool accessing a sub-site of the second site; when the routing table of the detached data pool is updated, read the data stripe to be migrated and the migration verification fragment of the detached data pool according to the metadata in the second metadata pool of the second site, wherein the metadata indicates the location of the data stripe to be migrated and the migration verification fragment in the storage device of the detached data pool; and verify the data stripe to be migrated sequentially according to the read migration verification fragment and the migration verification fragment stored in the second site, wherein the migration verification fragment stored in the second site was obtained through network transmission when the detached data pool was located at the first site.

[0100] In some embodiments of this disclosure, the data verification unit 710 is configured to: perform a first verification on the first data stripe to be migrated based on the read migration verification fragment; if the verification result of the first verification is a verification failure, then perform a second verification on the data stripe to be migrated using the migration verification fragment stored in the second site; if the verification result of the second verification is a verification success or the verification result of the first verification is a verification success, then determine that the first data stripe to be migrated has been successfully verified; if the verification result of the second verification is a verification failure, then determine that the first data stripe to be migrated has failed verification, and perform data recovery on the first data stripe to be migrated using the read migration verification fragment and the migration verification fragment stored in the second site.

[0101] It should be noted that the multi-site data transmission device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing multi-site data transmission. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the multi-site data transmission device can be divided into different program modules to complete all or part of the processing described above. In addition, the multi-site data transmission device provided in the above embodiments and the multi-site data transmission method provided in this disclosure belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0102] Figure 8 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 8 As shown, the electronic device 800 includes at least one processor 802; and a memory 801 communicatively connected to the at least one processor 802; wherein the memory 801 stores instructions executable by the at least one processor 802, the instructions being executed by the at least one processor 802 to implement the steps of the multi-site data transmission method described in the embodiments of this disclosure.

[0103] Optionally, the electronic device may specifically be a multi-site data transmission device according to the embodiments of this application, and the electronic device may implement the corresponding processes implemented by the multi-site data transmission device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0104] It is understood that the electronic device also includes a communication interface 803. Various components in the electronic device are coupled together via a bus system 804. It is understood that the bus system 804 is used to implement communication between these components. In addition to a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general labeled all buses as Bus System 804.

[0105] It is understood that memory 801 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 801 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0106] The methods disclosed in the above embodiments can be applied to or implemented by processor 802. Processor 802 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 802 or by instructions in software form. Processor 802 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 802 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 801. Processor 802 reads information from memory 801 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0107] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0108] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause the computer to execute the steps of the multi-site data transmission method described in this disclosure.

[0109] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-site data transmission method described in this disclosure.

[0110] Optionally, the computer-readable storage medium can be applied to the inter-site data transmission apparatus in the embodiments of this application, and the computer instructions cause the computer to execute the corresponding processes implemented by the inter-site data transmission apparatus in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0112] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0113] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0114] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0115] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0116] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of data transmission between multiple sites, characterized by, The method is applied to a first site, and the method includes: In response to a data copy command, all storage nodes in the original data pool of the first site are controlled to perform a data copy task, copying the data stripe to be migrated to a free data pool, which is located in a sub-site of the first site; When the data stripe to be migrated is completely copied to the free data pool, the free data pool is controlled to generate migration check fragments corresponding to the data stripe to be migrated according to the number of check fragments indicated by the erasure coding algorithm, and the data stripe to be migrated and the migration check fragments are stored in different storage devices in the free data pool respectively. The detached data pool is controlled to transmit the migration verification fragment to the second site via the network, so that the second site stores the migration verification fragment; Update the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronize the updated metadata to the second metadata pool of the second site. The updated metadata indicates the storage device location of the data stripe to be migrated and the migration verification shard in the detached data pool. The storage devices in the detached data pool are powered down to transport the detached data pool to the second site via physical transport, so that the second site can read the data stripe to be migrated and the migration verification fragment in the detached data pool and verify the data stripe to be migrated.

2. The multi-site data transfer method of claim 1, wherein, The step of responding to a data copy command and controlling all storage nodes in the original data pool of the first site to perform a data copy task, copying the data stripe to be migrated to the free data pool, includes: Obtain the historical performance value of the data copy task within a preset time period, and based on the historical performance value, predict the future performance value of the data copy task within the preset time period, so as to adjust the priority of the original data pool to execute the data copy task according to the future performance value.

3. A method of data transmission between multiple sites, characterized by, The method is applied to a second site, and the method includes: In response to the access of the storage device of the detached data pool to the sub-site of the second site, the routing table of the detached data pool, which is transported from the first site to the second site, is updated; When the routing table of the detached data pool is updated, the data stripe to be migrated and the migration verification fragment of the detached data pool are read according to the metadata in the second metadata pool of the second site. The metadata indicates the storage device location of the data stripe to be migrated and the migration verification fragment in the detached data pool. The data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the detached data pool in the first site. The first migration data stripe is subjected to a first verification based on the read migration verification fragment; If the first verification result is a verification failure, then the migration verification shard stored in the second site is used to perform a second verification on the first data stripe to be migrated. The migration verification shard stored in the second site is obtained through network transmission when the free data pool is located at the first site. If the verification result of the second verification is successful or the verification result of the first verification is successful, then the first data stripe to be migrated is determined to have been successfully verified. If the verification result of the second verification is a verification failure, it is determined that the first data stripe to be migrated has failed the verification, and the data of the first data stripe to be migrated is recovered using the read migration verification fragment and the migration verification fragment stored in the second site. If all the data stripes to be migrated are successfully verified, the detached state of the detached data pool will be adjusted to the normal state.

4. A multi-site data transmission system, characterized by The system includes: The first site management system located at the first site and the second site management system located at the second site; The first site management system is used to execute the multi-site data transmission method according to any one of claims 1 to 2; The second site management system is used to execute the inter-site data transmission method as described in claim 3.

5. A multi-site data transfer apparatus, characterized by comprising: The device is applied at a first site, and the device includes: The copy unit is used to respond to the data copy command and control all storage nodes in the original data pool of the first site to perform data copy tasks, copying the data stripe to be migrated to the free data pool, which is located in a sub-site of the first site; A verification shard generation unit is configured to, when the data stripe to be migrated is completely copied to the detached data pool, control the detached data pool to generate migration verification shards corresponding to the data stripe to be migrated according to the number of verification shards indicated by the erasure coding algorithm, and store the data stripe to be migrated and the migration verification shards in different storage devices in the detached data pool respectively; control the detached data pool to transmit the migration verification shards to a second site via a network, so that the second site stores the migration verification shards; update the metadata corresponding to the data stripe to be migrated in the first metadata pool of the first site, and synchronize the updated metadata to the second metadata pool of the second site, wherein the updated metadata indicates the storage device location of the data stripe to be migrated and the migration verification shards in the detached data pool; The migration unit is used to power down the storage devices in the detached data pool so as to transport the detached data pool to the second site through physical transportation, so that the second site can read the data stripe to be migrated and the migration verification fragment in the detached data pool and verify the data stripe to be migrated.

6. A multi-site data transfer apparatus, characterized by comprising: The device is applied at a second station, and the device includes: A data verification unit is configured to update the routing table of the detached data pool in response to the access of the storage device of the detached data pool to the sub-site of the second site. The detached data pool is transported from the first site to the second site. When the routing table of the detached data pool is updated, the unit reads the data stripe to be migrated and the migration verification fragment of the detached data pool according to the metadata in the second metadata pool of the second site. The metadata indicates the location of the data stripe to be migrated and the migration verification fragment in the storage device of the detached data pool. The data stripe to be migrated and the migration verification fragment are obtained through the original data pool and the detached data pool in the first site. The unit then verifies the first data stripe to be migrated based on the read migration verification fragment. First verification; if the first verification result is verification failure, then the first data stripe to be migrated is verified using the migration verification shard stored in the second site. The migration verification shard stored in the second site is obtained through network transmission when the free data pool is located at the first site; if the second verification result is verification success or the first verification result is verification success, then the first data stripe to be migrated is determined to have been verified successfully; if the second verification result is verification failure, then the first data stripe to be migrated is determined to have failed verification, and the first data stripe to be migrated is recovered using the read migration verification shard and the migration verification shard stored in the second site. The status adjustment unit is used to adjust the detached status of the detached data pool to the normal status if all the data stripes to be migrated are successfully verified.

7. An electronic device, comprising: include: The processor and the memory used to store computer programs that can run on the processor. When the processor runs the computer program, it executes the multi-site data transmission method according to any one of claims 1-2 or 3.

8. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the inter-site data transmission method according to any one of claims 1-2 or 3.

Images