A processing method of storing data and related apparatus

Abstract

The application discloses a processing method of storage data and a related device. The cold backup system corresponding to the storage data is determined based on the storage environment of the data source end, so that the cold backup data corresponding to the storage data is obtained through the cold backup system; then the cold backup data is transmitted from the data source end to the migration target end; and the migration target end is controlled to process the cold backup data to obtain processing data; and then the processing data is written into the migration target end according to a preset rule, and the preset rule is determined based on the storage environment of the migration target end. Therefore, the migration process of the storage data across clusters is efficient. Since the cold backup data is used as the data source for migration, the influence of the network state between the data source end and the migration target end on the data migration is avoided, and the storage system can be adapted to different storage systems, thereby improving the efficiency of the migration of the storage data across clusters.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for processing stored data and related apparatus. Background Technology

[0002] With the rapid development of internet technology, people have increasingly higher demands for data content, thus requiring corresponding storage systems to provide services. Cross-cluster data migration differs from intra-cluster data migration. Clusters are often interconnected, or experience high latency and poor quality. Furthermore, cross-cluster data migration typically involves migrating the entire business's data together, resulting in large data volumes and short migration time requirements, posing significant challenges to business operations and maintenance.

[0003] Generally, the cross-cluster data migration process involves traversing the data from the data migration source and then writing the traversed data to the data migration target.

[0004] However, the entire process of traversing data is arduous, complex, inflexible, and the migration tools have low reusability. For data migration tasks across regions or even continents, the high network latency makes the entire process quite time-consuming, affecting the efficiency of data migration across cluster storage. Summary of the Invention

[0005] In view of this, this application provides a method for processing stored data, which can effectively improve the efficiency of cross-cluster storage data migration.

[0006] The first aspect of this application provides a method for processing stored data, which can be applied to a system or program in a terminal device that includes data storage processing functions, specifically including:

[0007] Based on the storage environment configured at the data source, determine the cold backup system corresponding to the stored data, so as to obtain the cold backup data corresponding to the stored data through the cold backup system;

[0008] The cold backup data is transmitted from the data source to the migration target, where the storage cluster corresponding to the data source is different from the storage cluster corresponding to the migration target.

[0009] The target terminal of the migration is controlled to process the cold backup data to obtain processed data;

[0010] The processed data is written to the migration target terminal according to preset rules, which are determined based on the storage environment configured on the migration target terminal.

[0011] Optionally, in some possible implementations of this application, the step of transmitting the cold backup data from the data source to the migration target includes:

[0012] Obtain the network transmission parameters between the data source and the migration target.

[0013] The cold backup data is compressed based on the network transmission parameters to obtain compressed data;

[0014] The compressed data is divided according to a preset granularity to obtain multiple segmented data.

[0015] The split data is transmitted from the data source to the migration target via a concurrent channel.

[0016] Optionally, in some possible implementations of this application, the method further includes:

[0017] Monitor the network transmission parameters between the data source and the migration target.

[0018] If the network transmission parameters meet the transmission conditions, the cold backup data is segmented according to the preset segmentation granularity to obtain multiple transmission data.

[0019] The transmission data is transmitted from the data source to the migration target through the concurrent channel.

[0020] Optionally, in some possible implementations of this application, the step of transmitting the split data from the cold backup system to the migration target includes:

[0021] If the network transmission parameters indicate a network anomaly between the data source and the migration target, the split data is transmitted from the data source to the relay cluster.

[0022] The split data is transmitted to the migration target end through the relay cluster.

[0023] Optionally, in some possible implementations of this application, writing the processed data to the migration target terminal according to preset rules includes:

[0024] Compare the cluster distribution at the data source end with the cluster distribution at the migration target end;

[0025] If the cluster distribution at the data source end is the same as the cluster distribution at the migration target end, then the preset rule is determined to be a physical replacement method.

[0026] The processed data is written to the migration target using the physical replacement method;

[0027] or;

[0028] If the cluster distribution of the data source is the same as that of the target migration endpoint, then the write interface of the target migration endpoint is determined based on the preset rules.

[0029] The processed data is adapted to the target migration endpoint based on the write interface, so as to write the processed data to the target migration endpoint.

[0030] Optionally, in some possible implementations of this application, determining the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, and obtaining the cold backup data corresponding to the stored data through the cold backup system, includes:

[0031] In response to the bill of lading operation in the target interface, the storage environment configured on the data source end is obtained, and the storage environment configured on the migration target end is obtained based on the bill of lading operation.

[0032] The cold backup system corresponding to the stored data is determined based on the storage environment configured at the data source, so as to obtain the cold backup data corresponding to the stored data through the cold backup system;

[0033] The step of writing the processed data to the migration target terminal according to preset rules includes:

[0034] The preset rules are determined based on the storage environment configured at the data source and the storage environment configured at the migration target.

[0035] The processed data is written to the migration target terminal according to the preset rules.

[0036] Optionally, in some possible implementations of this application, the stored data includes a first file and a second file, and the method further includes:

[0037] The first file is obtained through the cold backup system;

[0038] Simultaneously, while transmitting the first file from the data source to the migration target, the second file is obtained through the cold backup system;

[0039] While controlling the target migration endpoint to process the first file to obtain the first file data, the second file is transmitted from the data source endpoint to the target migration endpoint;

[0040] While writing the first file data to the migration target terminal according to the preset rules, the migration target terminal is controlled to process the second file to obtain the second file data;

[0041] The second file data is written to the migration target terminal according to the preset rules.

[0042] A second aspect of this application provides a data storage processing apparatus, comprising:

[0043] The acquisition unit is used to determine the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, so as to obtain the cold backup data corresponding to the stored data through the cold backup system;

[0044] A transmission unit is used to transmit the cold backup data from the data source to the migration target, wherein the storage cluster corresponding to the data source is different from the storage cluster corresponding to the migration target.

[0045] The processing unit is used to control the migration target terminal to process the cold backup data to obtain processed data;

[0046] The processing unit is further configured to write the processed data to the migration target terminal according to a preset rule, wherein the preset rule is determined based on the storage environment configured on the migration target terminal.

[0047] Optionally, in some possible implementations of this application, the transmission unit is specifically used to obtain network transmission parameters between the data source and the migration target.

[0048] The transmission unit is specifically used to compress the cold backup data based on the network transmission parameters to obtain compressed data;

[0049] The transmission unit is specifically used to divide the compressed data according to a preset division granularity to obtain multiple divided data.

[0050] The transmission unit is specifically used to transmit the split data from the data source to the migration target via a concurrent channel.

[0051] Optionally, in some possible implementations of this application, the transmission unit is specifically used to monitor the network transmission parameters between the data source and the migration target.

[0052] The transmission unit is specifically used to divide the cold backup data according to the preset division granularity to obtain multiple transmission data if the network transmission parameters meet the transmission conditions.

[0053] The transmission unit is specifically used to transmit the transmission data from the data source to the migration target through the concurrent channel.

[0054] Optionally, in some possible implementations of this application, the transmission unit is specifically used to transmit the split data from the data source to the relay cluster if the network transmission parameters indicate a network anomaly between the data source and the migration target.

[0055] The transmission unit is specifically used to transmit the split data to the migration target end through the relay cluster.

[0056] Optionally, in some possible implementations of this application, the processing unit is specifically used to compare the cluster distribution of the data source end with the cluster distribution of the migration target end;

[0057] The processing unit is specifically used to determine that the preset rule is a physical replacement method if the cluster distribution at the data source end is the same as the cluster distribution at the migration target end.

[0058] The processing unit is specifically used to write the processed data into the migration target terminal using the physical replacement method.

[0059] or;

[0060] The processing unit is specifically used to determine the write interface of the migration target end based on the preset rules if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end.

[0061] The processing unit is specifically used to adapt the processed data to the migration target end based on the write interface, so as to write the processed data to the migration target end.

[0062] Optionally, in some possible implementations of this application, the acquisition unit is specifically used to acquire the storage environment configured on the data source end in response to the bill of lading operation on the target interface, and to acquire the storage environment configured on the migration target end based on the bill of lading operation.

[0063] The acquisition unit is specifically used to determine the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, so as to acquire the cold backup data corresponding to the stored data through the cold backup system.

[0064] The processing unit is specifically used to determine the preset rules based on the storage environment configured at the data source end and the storage environment configured at the migration target end.

[0065] The processing unit is specifically used to write the processed data into the migration target terminal according to the preset rules.

[0066] Optionally, in some possible implementations of this application, the processing unit is specifically used to obtain the first file through the cold backup system;

[0067] The processing unit is specifically used to transfer the first file from the data source to the migration target while simultaneously obtaining the second file through the cold backup system.

[0068] The processing unit is specifically used to control the migration target end to process the first file to obtain the first file data, while simultaneously transmitting the second file from the data source end to the migration target end;

[0069] The processing unit is specifically used to write the first file data to the migration target terminal according to the preset rules, while controlling the migration target terminal to process the second file to obtain the second file data;

[0070] The processing unit is specifically used to write the second file data to the migration target terminal according to the preset rules.

[0071] A third aspect of this application provides a computer device, comprising: a memory, a processor, and a bus system; the memory is used to store program code; the processor is used to execute the data processing method described in the first aspect or any one of the first aspects according to instructions in the program code.

[0072] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the data processing method described in the first aspect or any one of the first aspects.

[0073] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method provided in the first aspect or various optional implementations thereof.

[0074] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0075] By determining the corresponding cold backup system based on the storage environment configured at the data source end, and obtaining the corresponding cold backup data from the cold backup system, the cold backup data is then transferred from the data source end to the migration target end. The storage cluster corresponding to the data source end is different from that of the migration target end. The migration target end is then controlled to process the cold backup data to obtain processed data. Finally, the processed data is written to the migration target end according to preset rules, which are determined based on the storage environment configured at the migration target end. This efficient cross-cluster data migration process avoids the impact of network status between the data source end and the migration target end on data migration because it uses cold backup data as the data source for migration. It can also be adapted to different storage systems, improving the efficiency of cross-cluster data migration. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0077] Figure 1 A network architecture diagram for a data processing system that stores and processes data.

[0078] Figure 2 A flowchart illustrating the processing of stored data is provided in an embodiment of this application.

[0079] Figure 3 A flowchart illustrating a data storage processing method provided in this application embodiment;

[0080] Figure 4 A schematic diagram illustrating a data storage processing method provided in an embodiment of this application;

[0081] Figure 5 A schematic diagram illustrating a scenario for another data storage processing method provided in an embodiment of this application;

[0082] Figure 6 A schematic diagram illustrating a scenario for another data storage processing method provided in an embodiment of this application;

[0083] Figure 7 A schematic diagram illustrating a scenario for another data storage processing method provided in an embodiment of this application;

[0084] Figure 8 A schematic diagram illustrating a scenario for another data storage processing method provided in an embodiment of this application;

[0085] Figure 9 A schematic diagram illustrating a scenario for another data storage processing method provided in an embodiment of this application;

[0086] Figure 10 A flowchart illustrating another data storage processing method provided in this application embodiment;

[0087] Figure 11 This is a schematic diagram of the structure of a data storage processing device provided in an embodiment of this application;

[0088] Figure 12 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0089] Figure 13This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0090] This application provides a method and related apparatus for processing stored data, which can be applied to systems or programs in terminal devices that include stored data processing functions. The method involves determining a cold backup system corresponding to the stored data based on the storage environment configured at the data source end, and then obtaining the cold backup data corresponding to the stored data through the cold backup system. The cold backup data is then transmitted from the data source end to the migration target end, where the storage cluster corresponding to the data source end is different from that of the migration target end. The migration target end is then controlled to process the cold backup data to obtain processed data. Finally, the processed data is written to the migration target end according to preset rules, which are determined based on the storage environment configured at the migration target end. This enables efficient cross-cluster migration of stored data. Because cold backup data is used as the data source for migration, the influence of network status between the data source end and the migration target end on data migration is avoided, and the method can be adapted to different storage systems, improving the efficiency of cross-cluster migration of stored data.

[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0092] First, some terms that may appear in the embodiments of this application will be explained.

[0093] Cold backup: also known as offline backup, refers to a full backup of the database performed when the database is shut down and cannot be updated, and allows for specified data recovery.

[0094] Cluster: A cluster system is a computer system that works in close collaboration through a loosely integrated set of computer software and / or hardware to perform computational tasks.

[0095] Dump: At a specific time, the contents of the entire storage device or a part of the storage device are recorded to another storage device.

[0096] It should be understood that the data processing method provided in this application can be applied to systems or programs in terminal devices that include data processing functions, such as database management applications. Specifically, the data processing system can run on systems such as... Figure 1 In the network architecture shown, such as Figure 1 The diagram shows the network architecture of the data storage processing system. As can be seen, the system can process stored data from multiple information sources. This involves initiating data migration through target operations on the terminal side, enabling data sharing between servers across the cluster. It can be understood that... Figure 1 The document illustrates various terminal devices, which can be computer devices. In real-world scenarios, more or fewer types of terminal devices may participate in the data storage and processing process. The specific number and types depend on the actual scenario and are not limited here. Figure 1 The image shows one server, but in real-world scenarios, multiple servers can participate; the specific number of servers depends on the actual scenario.

[0097] In this embodiment, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and the terminal and server can be connected to form a blockchain network; this application does not impose any restrictions.

[0098] It is understood that the aforementioned data storage processing system can run on personal mobile terminals, such as database management applications, or on servers, or on third-party devices to provide data storage processing to obtain the processing results of the stored data from the information source. Specifically, the data storage processing system can run as a program on the aforementioned devices, or as a system component within the aforementioned devices, or as a cloud service program. The specific operating mode depends on the actual scenario and is not limited here.

[0099] With the rapid development of internet technology, people have increasingly higher demands for data content, thus requiring corresponding storage systems to provide services. Cross-cluster data migration differs from intra-cluster data migration. Clusters are often interconnected, or experience high latency and poor quality. Furthermore, cross-cluster data migration typically involves migrating the entire business's data together, resulting in large data volumes and short migration time requirements, posing significant challenges to business operations and maintenance.

[0100] Generally, the cross-cluster data migration process involves traversing the data from the data migration source and then writing the traversed data to the data migration target.

[0101] However, the entire process of traversing data is arduous, complex, inflexible, and the migration tools have low reusability. For data migration tasks across regions or even continents, the high network latency makes the entire process quite time-consuming, affecting the efficiency of data migration across cluster storage.

[0102] Therefore, for storage systems, implementing an efficient cross-cluster data migration solution presents the following challenges: First, the data source machines are numerous and the data types are diverse, making management and statistics difficult; second, record-level data migration efficiency is low when network quality is poor; third, there are no readily available tools, making cross-system data migration difficult; and fourth, the entire process is complex, and manual or script-based operations are prone to errors. The following embodiment will elaborate on solutions to these four challenges.

[0103] To address the aforementioned problems, this application proposes a method for processing stored data, which is applied to... Figure 2 In the workflow framework for processing stored data shown, such as Figure 2 The diagram shown is a flowchart of a data storage processing process provided in an embodiment of this application. Through data processing operations performed on the terminal, cluster 1 (source data end) obtains real-time cold backup data, compresses and splits the cold backup data, and transmits it to cluster 2 (migration target end). After cluster 2 processes and adapts the cold backup data, it can be written to cluster 2, thereby realizing cross-cluster migration of stored data.

[0104] This process framework is compatible with various existing cloud backup systems for obtaining data sources and supports rapid cross-continental data migration globally. It also supports cross-system data migration, such as migrating data to MySQL or Redis. Furthermore, it supports completing the entire process through a page-based order submission, reducing operational costs. These steps are described in the following description.

[0105] It is understood that the method provided in this application can be a program written as processing logic in a hardware system, or a data storage processing device, implemented in an integrated or external manner. As one implementation, the data storage processing device determines the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, and obtains the cold backup data corresponding to the stored data through the cold backup system; then, it transmits the cold backup data from the data source end to the migration target end, where the storage cluster corresponding to the data source end is different from the storage cluster corresponding to the migration target end; it controls the migration target end to process the cold backup data to obtain processed data; and then writes the processed data to the migration target end according to preset rules, which are determined based on the storage environment configured at the migration target end. This results in an efficient cross-cluster data migration process. Because cold backup data is used as the data source for migration, the influence of network status between the data source end and the migration target end on data migration is avoided, and it can be adapted to different storage systems, improving the efficiency of cross-cluster data migration.

[0106] Based on the above process architecture, the data processing method in this application will be described below. Please refer to [link / reference]. Figure 3 , Figure 3 The flowchart illustrates a data storage processing method provided in this application embodiment. This management method can be executed by a participant, and this application embodiment includes at least the following steps:

[0107] 301. Determine the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, so as to obtain the cold backup data corresponding to the stored data through the cold backup system.

[0108] Understandably, data migration is a crucial and irreplaceable operational function for any storage system. This embodiment can be applied to the KV database TcaplusDB. TcaplusDB utilizes this embodiment for data migration during its daily operations, such as global intercontinental data migration and copying and migrating data from the production environment to a data analysis system.

[0109] Specifically, since distributed storage systems may be deployed in multiple regions and environments, obtaining cold backup data requires strong compatibility with these scenarios to ensure successful cold backup. Therefore, it's necessary to determine the corresponding cold backup system based on the storage environment configured at the data source. In one possible scenario, for a distributed storage system deployed on a company's intranet, a proprietary cold backup system or an open-source cold backup system like Gluster can be used. For distributed storage systems deployed in cloud environments, the cloud's own cold backup system is generally preferred. For example, for a distributed storage system deployed on AWS, AWS S3 can be chosen as the cold backup system. For environments without a cold backup system, it is recommended to directly dump the cold backup data from the relevant storage nodes in real time.

[0110] Therefore, this embodiment can automatically be compatible with the following four methods of obtaining cold backups: firstly, it is compatible with open-source cold backup systems in the industry, such as Gluster; secondly, it is compatible with customized cold backup systems, such as AWS S3; thirdly, it is compatible with proprietary (customized) cold backup systems; in addition, it dumps the cold backup in real time on the storage node when there is no cold backup, thereby improving the adaptability of cold backup data acquisition during data migration.

[0111] 302. Transfer the cold backup data from the data source to the migration target.

[0112] In this embodiment, the storage cluster corresponding to the data source is different from the storage cluster corresponding to the migration target. That is, this embodiment can be applied to the cross-cluster data migration process. The specific cluster span can be different geographical locations, such as cross-continental data migration; or different network states, such as cross-shielded scenario data migration. The specific cluster distribution depends on the actual scenario.

[0113] Specifically, when transmitting cold backup data, it can be compressed and segmented according to the real-time network status. First, the network transmission parameters between the data source and the migration target are obtained, such as transmission rate, network speed, and network latency. Then, the cold backup data is compressed based on the network transmission parameters to obtain compressed data. For example, the slower the transmission rate, the smaller the compressed cold backup data is. The compressed data is then segmented according to a preset segmentation granularity to obtain multiple segmented data. Finally, the segmented data is transmitted from the data source to the migration target through a concurrent channel.

[0114] In one possible scenario, after obtaining the cold backup, the data can be compressed and split into roughly equal file blocks, depending on the situation. If subsequent cold backup files need to be transferred across continents or where transmission efficiency is low, a highly efficient compression algorithm should be chosen for the files. File compression can be achieved using common tools such as lzo and tar.

[0115] Understandably, since cold backup files vary in size, in order to improve subsequent transmission efficiency, cold backup files can be divided into file blocks of roughly the same size. For example, all cold backup files can be divided into file blocks of 512MB each, with the specific size depending on the specific situation.

[0116] Additionally, if subsequent cold backup file transfer efficiency is high, or even if transfer is not a primary concern, compression can be avoided. This involves monitoring network transmission parameters between the data source and the migration target; if these parameters meet the transmission requirements, the cold backup data is split according to a preset granularity to obtain multiple transfer data sets; then, these data are transmitted from the data source to the migration target via concurrent channels, thereby improving the processing efficiency of the cold backup data.

[0117] In one possible scenario, the acquisition of cold backup data is performed in real time, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of a data storage processing method provided in an embodiment of this application; the diagram shows the execution control of daily cold backup, thereby ensuring the integrity and matching degree of the cold backup data.

[0118] In another possible scenario, since cold backup files all need to be transferred, the networks of the source and destination endpoints of the data migration may be unreachable (e.g., network blocked) or have very poor communication quality. Transferring cold backup files is a major time-consuming part of the data migration task, especially for intercontinental data migration scenarios, where efficient transfer of cold backup files is paramount to improving efficiency. For scenarios where the source and destination endpoints of the data migration are unreachable, a relay cluster can be used to complete this transfer task. That is, if network transmission parameters indicate a network anomaly between the data source and the migration destination, the split data is transferred from the data source to the relay cluster; then, the relay cluster sends the split data to the migration destination.

[0119] Understandably, network communication quality between a relay cluster and both the source and destination ends is better than direct communication between the source and destination ends. Of course, if there is no relay cluster, or if the network quality between the source and destination ends is acceptable, cold backup files can be directly transferred from the data migration source end to the data migration destination end. Standard tools such as scp can be used to transfer cold backup files.

[0120] In this embodiment, a pipelined transmission of cold backup files can also be employed. That is, cold backup files are generated and transmitted simultaneously, parallelizing the generation and transmission of cold backup files. Furthermore, adjusting the operating system's network transmit / receive buffers can also improve file transfer efficiency. For example, in Linux systems, `net.core.wmem_max` and `net.core.rmem_max` can be increased to over 20MB in the ` / etc / sysctl.conf` file. Moreover, to fully utilize network bandwidth, multiple channels should be used for concurrent file transmission. Because a file is multi-splitted, even with a small number of cold backup files, transmission efficiency can be improved several times over.

[0121] The above embodiments directly use cold backup as the data migration source, first remotely copying the cold backup data files to the same region as the target storage system, ensuring the feasibility of the migration process; and can import data to the target end through various methods; and can also migrate data to various different storage systems; and supports completing the entire process by submitting a form on the page. This effect can be described in conjunction with the following embodiments.

[0122] 303. Control the target end of the migration to process the cold backup data to obtain processed data.

[0123] In this embodiment, since the storage system or cluster distribution of the migration target may be different from that of the source data end, data processing is required.

[0124] It is understandable that compression and splitting may have occurred during the cold backup data transmission process. At this time, it is necessary to decompress the data accordingly and write it in combination with the specific storage system or cluster distribution.

[0125] 304. Write the processed data to the migration target end according to the preset rules.

[0126] In this embodiment, the preset rules are determined based on the storage environment configured at the migration target end, that is, the difference between the storage system (storage environment) at the data source end and the storage system (storage environment) at the migration target end. Since this embodiment can be applied to cross-cluster data migration, and clusters have the characteristic of node distribution, the difference between the storage system at the data source end and the storage system at the migration target end can be compared by comparing the cluster distribution at the data source end and the cluster distribution at the migration target end.

[0127] Specifically, after the cold backup file is transferred to the data migration target, there are multiple methods to apply the data from the cold backup file to the target, and the target supports various storage systems. For different pre-defined rules, data import can be achieved by physically replacing the cold backup file in a completely new environment on the target; or by using the target storage system API to write the data to the target.

[0128] In one possible scenario, assuming the data migration target is a completely new storage system, the routing distribution and other deployment details of the source storage system can be fully replicated. First, the cluster distribution of the data source and the target storage system are compared. If the cluster distributions are identical, the default rule is physical replacement. Then, the processed data is written to the target storage system using physical replacement. This allows the cold backup file to be directly placed into the target storage system for immediate use, saving time and effort. A diagram illustrating the physical replacement cold backup file import scheme is shown below. Figure 5 As shown, Figure 5 This is a schematic diagram of another data storage processing method provided in the embodiments of this application; the diagram shows that the compressed and split cold backup files are first merged and decompressed, and then the corresponding cluster nodes (deployment details such as the routing distribution of the storage system) are directly physically replaced to achieve efficient data writing.

[0129] Additionally, for scenarios where the target storage system is an existing system, it may be the same as the source system or a different system. One approach to importing data to the target system is to use a tool to read all data from the cold backup file, then process this data (if needed) and write it to the target storage system via its API. A diagram illustrating the target-side API data import approach is shown below. Figure 6 As shown, Figure 6 This is a schematic diagram of another data storage processing method provided in this application embodiment; that is, if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end, the write interface of the migration target end is determined based on preset rules; then the processed data is adapted to the migration target end based on the write interface, so as to write the processed data to the migration target end, thereby realizing the write adaptation process of multiple storage systems.

[0130] In one possible embodiment, the entire data migration process described above can support pipelined operations, fully utilizing parallelism to shorten operation time and improve efficiency. When server resources are limited, server and network resources can be used efficiently, allowing each step to run in parallel. For example, in a scenario where the stored data includes a first file and a second file, the first file can be obtained first through a cold backup system; then, while transferring the first file from the data source to the migration target, the second file can be obtained through the cold backup system; and while controlling the migration target to process the first file to obtain its data, the second file can be transferred from the data source to the migration target; then, according to preset rules, the first file data can be written to the migration target while controlling the migration target to process the second file to obtain its data; finally, according to preset rules, the second file data can be written to the migration target.

[0131] In another possible scenario, such as Figure 7 As shown, Figure 7 This diagram illustrates a scenario for another data storage processing method provided in this application embodiment. The diagram shows five tables (files) A, B, C, D, and E that need to be migrated. First, table A obtains a cold backup from the cold backup center. Then, while table A is being transferred for cold backup, table B can also obtain a cold backup from the cold backup center. Next, table A is imported into the target server, table B can be transferred for cold backup, table C can obtain a cold backup from the cold backup center, and so on. Of course, the pipeline operation unit depends on the server situation. For example, if there are sufficient server resources, tables A and B can be obtained for cold backup together from the cold backup center, and subsequently transferred for cold backup data together. That is, the pipeline operation unit can operate on several tables simultaneously. The specific number of files depends on the actual scenario and is not limited here.

[0132] Understandably, data migration is a complex and demanding task. Even with a clear understanding of all processes, manual or script-based operations are not only time-consuming and labor-intensive but also prone to errors. This embodiment uses a page-based order submission to complete the entire task. Since the source and destination ends of cross-cluster data migration are typically two completely isolated systems, tasks need to be generated in both systems: the source end constructs the data, and the destination end imports the data. Alternatively, if the source and destination ends are the same system, the entire process can be completed directly through a page-based order submission within that system. Specifically, the process responds to the order submission on the target interface to obtain the storage environment configured on the data source end and the storage environment configured on the migration target end. Then, based on the storage environment configured on the data source end, the corresponding cold backup system is determined, and the corresponding cold backup data is obtained through the cold backup system. For the subsequent writing process, preset rules are determined based on the storage environment configured on the data source end and the storage environment configured on the migration target end. Finally, the processed data is written to the migration target end according to the preset rules.

[0133] In one possible scenario, the diagram illustrating the entire data migration task completed by the page system's order submission is shown below. Figure 8 As shown, Figure 8 This is a schematic diagram illustrating another data storage processing method provided in this application embodiment; through a bill of lading operation on a browser page, the system adaptation process of the source and target ends of data migration is triggered simultaneously, thereby automatically executing the data migration process described in the above embodiment. Specifically, the bill of lading operation can be performed using... Figure 9 The interface shown is as follows: Figure 9This is a schematic diagram of another data storage processing method provided in the embodiments of this application; the figure shows the browser interface, where relevant personnel can select the data source and migration target (cluster selection) for data migration, and then click on migration order A1 to automatically perform the data migration process described in the above embodiments, which will not be elaborated here.

[0134] As described in the above embodiments, the cold backup system corresponding to the stored data is determined based on the storage environment configured at the data source end, and the cold backup data corresponding to the stored data is obtained through the cold backup system. Then, the cold backup data is transmitted from the data source end to the migration target end, where the storage cluster corresponding to the data source end is different from that of the migration target end. The migration target end is then controlled to process the cold backup data to obtain processed data. Finally, the processed data is written to the migration target end according to preset rules, which are determined based on the storage environment configured at the migration target end. This efficient cross-cluster data migration process, by using cold backup data as the data source, avoids the impact of network conditions between the data source end and the migration target end on data migration, and can be adapted to different storage systems, thus improving the efficiency of cross-cluster data migration.

[0135] Therefore, after using this embodiment, storage system administrators can be freed from complex and cumbersome data migration tasks and complete the entire function directly by submitting an order on the page. It can be applied not only to large and complex scenarios such as global cross-continental data migration; it can also be applied to data migration across storage systems, enabling data to be reused in multiple scenarios; and it can also copy and migrate data from one environment to another for analysis and problem localization.

[0136] The above-mentioned functions play an important role in multiple scenarios. For example, in gaming, cross-server player battles require data migration, which is explained below. Please refer to [link / reference]. Figure 10 , Figure 10 A flowchart illustrating another data storage processing method provided in this application embodiment, which includes at least the following steps:

[0137] 1001. Obtain the user information corresponding to the target user in the first game server.

[0138] In this embodiment, the first game server can be a server frequently used by players. Since there are some cross-server activities in the game (such as global competitions), it is necessary to migrate game data across servers to ensure the smooth and normal operation of the game business.

[0139] 1002. Retrieve the cold backup data corresponding to the user information.

[0140] In this embodiment, the cold backup data corresponding to the user information can be the cold backup data that was last automatically updated, such as the data updated in the last 12 hours or the data updated after the target user last logged in. The specific method depends on the actual scenario.

[0141] 1003. Transfer the cold backup data to the second game server.

[0142] In this embodiment, the cold backup data transmission process is described in detail below. Figure 3 Step 303 of the illustrated embodiment will not be described in detail here.

[0143] 1004. Provide game services to target users based on a second game server.

[0144] In this embodiment, the process of writing cold backup data to the second game server is described in [reference needed]. Figure 3 Step 304 of the illustrated embodiment will not be described in detail here.

[0145] After the cold backup data is written to the second game server, the second game server can be used by players to log in and provide the same game services as the first server.

[0146] It is understood that after the game activity ends, the method of this embodiment can also be used to migrate the target user's game data (after the second game server) to the first server. The specific number of migrations depends on the actual scenario, which will not be elaborated here.

[0147] The above embodiments ensure the normal operation of cross-server games and can be adapted to game operation systems in different regions, ensuring the efficiency and effectiveness of game data migration.

[0148] To better implement the above-described solutions of the embodiments of this application, related apparatus for implementing the above solutions is also provided below. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of a data storage processing device provided in an embodiment of this application. The data storage processing device 1100 includes:

[0149] The acquisition unit 1101 is used to determine the cold backup system corresponding to the stored data based on the storage environment configured at the data source end, so as to acquire the cold backup data corresponding to the stored data through the cold backup system.

[0150] Transmission unit 1102 is used to transmit the cold backup data from the data source end to the migration target end, wherein the storage cluster corresponding to the data source end is different from the storage cluster corresponding to the migration target end.

[0151] Processing unit 1103 is used to control the migration target end to process the cold backup data to obtain processed data;

[0152] The processing unit 1103 is further configured to write the processed data to the migration target terminal according to a preset rule, wherein the preset rule is determined based on the storage environment configured on the migration target terminal.

[0153] Optionally, in some possible implementations of this application, the transmission unit 1102 is specifically used to obtain network transmission parameters between the data source and the migration target.

[0154] The transmission unit 1102 is specifically used to compress the cold backup data based on the network transmission parameters to obtain compressed data;

[0155] The transmission unit 1102 is specifically used to divide the compressed data according to a preset division granularity to obtain multiple divided data.

[0156] The transmission unit 1102 is specifically used to transmit the split data from the data source end to the migration target end through a concurrent channel.

[0157] Optionally, in some possible implementations of this application, the transmission unit 1102 is specifically used to monitor the network transmission parameters between the data source and the migration target.

[0158] The transmission unit 1102 is specifically used to divide the cold backup data according to the preset division granularity to obtain multiple transmission data if the network transmission parameters meet the transmission conditions.

[0159] The transmission unit 1102 is specifically used to transmit the transmission data from the data source end to the migration target end through the concurrent channel.

[0160] Optionally, in some possible implementations of this application, the transmission unit 1102 is specifically used to transmit the split data from the data source to the relay cluster if the network transmission parameters indicate a network anomaly between the data source and the migration target.

[0161] The transmission unit 1102 is specifically used to transmit the split data to the migration target end through the relay cluster.

[0162] Optionally, in some possible implementations of this application, the processing unit 1103 is specifically used to compare the cluster distribution of the data source end with the cluster distribution of the migration target end;

[0163] The processing unit 1103 is specifically used to determine that the preset rule is a physical replacement method if the cluster distribution at the data source end is the same as the cluster distribution at the migration target end.

[0164] The processing unit 1103 is specifically used to write the processed data into the migration target terminal using the physical replacement method;

[0165] or;

[0166] The processing unit 1103 is specifically used to determine the write interface of the migration target end based on the preset rules if the cluster distribution of the data source end is the same as the cluster distribution of the migration target end.

[0167] The processing unit 1103 is specifically used to adapt the processed data to the migration target end based on the writing interface, so as to write the processed data to the migration target end.

[0168] Optionally, in some possible implementations of this application, the acquisition unit 1101 is specifically used to acquire the storage environment configured on the data source end in response to the bill of lading operation on the target interface, and to acquire the storage environment configured on the migration target end based on the bill of lading operation.

[0169] The acquisition unit 1101 is specifically used to determine the cold backup system corresponding to the stored data according to the storage environment configured at the data source end, so as to obtain the cold backup data corresponding to the stored data through the cold backup system.

[0170] The processing unit 1103 is specifically used to determine the preset rule based on the storage environment configured at the data source end and the storage environment configured at the migration target end;

[0171] The processing unit 1103 is specifically used to write the processed data into the migration target terminal according to the preset rules.

[0172] Optionally, in some possible implementations of this application, the processing unit 1103 is specifically used to obtain the first file through the cold backup system;

[0173] The processing unit 1103 is specifically used to transfer the first file from the data source to the migration target while simultaneously obtaining the second file through the cold backup system.

[0174] The processing unit 1103 is specifically used to control the migration target end to process the first file to obtain the first file data, while simultaneously transmitting the second file from the data source end to the migration target end;

[0175] The processing unit 1103 is specifically used to write the first file data to the migration target terminal according to the preset rules, while controlling the migration target terminal to process the second file to obtain the second file data;

[0176] The processing unit 1103 is specifically used to write the second file data to the migration target terminal according to the preset rules.

[0177] By determining the corresponding cold backup system based on the storage environment configured at the data source end, and obtaining the corresponding cold backup data from the cold backup system, the cold backup data is then transferred from the data source end to the migration target end. The storage cluster corresponding to the data source end is different from that of the migration target end. The migration target end is then controlled to process the cold backup data to obtain processed data. Finally, the processed data is written to the migration target end according to preset rules, which are determined based on the storage environment configured at the migration target end. This efficient cross-cluster data migration process avoids the impact of network status between the data source end and the migration target end on data migration because it uses cold backup data as the data source for migration. It can also be adapted to different storage systems, improving the efficiency of cross-cluster data migration.

[0178] This application also provides a terminal device, such as... Figure 12 The diagram shown is a structural schematic of another terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0179] Figure 12 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 12 The mobile phone includes components such as a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a processor 1280, and a power supply 1290. Those skilled in the art will understand that... Figure 12 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0180] The following is combined with Figure 12 A detailed introduction to each component of a mobile phone:

[0181] RF circuit 1210 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 1280; additionally, it transmits uplink data to the base station. Typically, RF circuit 1210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 1210 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0182] The memory 1220 can be used to store software programs and modules. The processor 1280 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1220. The memory 1220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0183] The input unit 1230 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232. The touch panel 1231, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1231, as well as air touch operations within a certain range on the touch panel 1231), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1231 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 1280, and can receive and execute commands sent by the processor 1280. In addition, the touch panel 1231 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides the touch panel 1231, the input unit 1230 may also include other input devices 1232. Specifically, other input devices 1232 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0184] The display unit 1240 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1240 may include a display panel 1241, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, a touch panel 1231 may cover the display panel 1241. When the touch panel 1231 detects a touch operation on or near it, it transmits the information to the processor 1280 to determine the type of touch event. Subsequently, the processor 1280 provides corresponding visual output on the display panel 1241 according to the type of touch event. Although in Figure 12 In this embodiment, the touch panel 1231 and the display panel 1241 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1231 and the display panel 1241 can be integrated to realize the input and output functions of the mobile phone.

[0185] The mobile phone may also include at least one sensor 1250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1241 according to the ambient light level, and the proximity sensor can turn off the display panel 1241 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0186] Audio circuit 1260, speaker 1261, and microphone 1262 provide an audio interface between the user and the mobile phone. Audio circuit 1260 converts received audio data into electrical signals and transmits them to speaker 1261, where speaker 1261 converts them into sound signals for output. On the other hand, microphone 1262 converts collected sound signals into electrical signals, which are received by audio circuit 1260, converted into audio data, and then processed by processor 1280 before being transmitted via RF circuit 1210 to, for example, another mobile phone, or the audio data can be output to memory 1220 for further processing.

[0187] WiFi is a short-range wireless transmission technology. Through the WiFi module 1270, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 12 The WiFi module 1270 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0188] The processor 1280 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 1220, and by calling data stored in the memory 1220. Optionally, the processor 1280 may include one or more processing units; optionally, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 1280.

[0189] The mobile phone also includes a power supply 1290 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1280 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0190] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0191] In this embodiment of the application, the processor 1280 included in the terminal also has the function of performing the various steps of the page processing method described above.

[0192] This application also provides a server; please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations in the storage media 1330 on the server 1300.

[0193] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0194] The steps performed by the management device in the above embodiments can be based on this Figure 13 The server structure shown.

[0195] This application also provides a computer-readable storage medium storing data processing instructions, which, when executed on a computer, cause the computer to perform the aforementioned actions. Figures 3 to 10 The steps performed by the data storage processing device in the method described in the illustrated embodiment.

[0196] This application also provides a computer program product including data storage processing instructions, which, when run on a computer, causes the computer to perform the aforementioned actions. Figures 3 to 10 The steps performed by the data storage processing device in the method described in the illustrated embodiment.

[0197] This application embodiment also provides a data storage processing system, which may include... Figure 11 The data processing device described in the embodiments, or Figure 12 The terminal device in the described embodiments, or Figure 13 The server described.

[0198] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0200] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0202] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a data processing device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing stored data, characterized in that, Applications include cross-continental data migration scenarios where the source and destination networks are unavailable or have poor communication quality, including: In response to the bill of lading operation on the target interface, the storage environment configured on the data source side is obtained, and the storage environment configured on the migration target side is obtained based on the bill of lading operation. The cold backup system corresponding to the stored data is determined based on the storage environment configured at the data source end, so as to obtain the cold backup data corresponding to the stored data through the cold backup system; Obtain the network transmission parameters between the data source and the migration target, including transmission rate, network speed, and network latency; The cold backup data is compressed based on the network transmission parameters to obtain compressed data; The compressed data is divided according to a preset granularity to obtain multiple segmented data. The split data is transmitted from the data source to the migration target via a concurrent channel, wherein the storage cluster corresponding to the data source is different from the storage cluster corresponding to the migration target. The target migration endpoint is controlled to process the segmented data to obtain processed data, the processing including decompression; The process involves writing the processed data to the migration target terminal according to a preset rule, including: determining the preset rule based on the storage environment configured on the data source terminal and the storage environment configured on the migration target terminal; and writing the processed data to the migration target terminal according to the preset rule, wherein the preset rule is determined based on the storage environment configured on the migration target terminal. The step of transmitting the segmented data from the cold backup system to the migration target includes: If the network transmission parameters indicate a network anomaly between the data source and the migration target, the split data is transmitted from the data source to the relay cluster. The split data is transmitted to the migration target end through the relay cluster.

2. The method according to claim 1, characterized in that, The method further includes: Monitor the network transmission parameters between the data source and the migration target. If the network transmission parameters meet the transmission conditions, the cold backup data is segmented according to the preset segmentation granularity to obtain multiple transmission data. The transmission data is transmitted from the data source to the migration target through the concurrent channel.

3. The method according to claim 1, characterized in that, The step of writing the processed data to the migration target terminal according to preset rules includes: Compare the cluster distribution at the data source end with the cluster distribution at the migration target end; If the cluster distribution at the data source end is the same as the cluster distribution at the migration target end, then the preset rule is determined to be a physical replacement method. The processed data is written to the migration target using the physical replacement method; or; If the cluster distribution of the data source is the same as that of the target migration endpoint, then the write interface of the target migration endpoint is determined based on the preset rules. The processed data is adapted to the target migration endpoint based on the write interface, so as to write the processed data to the target migration endpoint.

4. The method according to any one of claims 1-3, characterized in that, The stored data includes a first file and a second file, and the method further includes: The first file is obtained through the cold backup system; Simultaneously, while transmitting the first file from the data source to the migration target, the second file is obtained through the cold backup system; While controlling the target migration endpoint to process the first file to obtain the first file data, the second file is transmitted from the data source endpoint to the target migration endpoint; While writing the first file data to the migration target terminal according to the preset rules, the migration target terminal is controlled to process the second file to obtain the second file data; The second file data is written to the migration target terminal according to the preset rules.

5. A data processing apparatus, characterized in that, Applications include cross-continental data migration scenarios where the source and destination networks are unavailable or have poor communication quality, including: The acquisition unit is used to acquire the storage environment configured on the data source end in response to the bill of lading operation on the target interface, and acquire the storage environment configured on the migration target end based on the bill of lading operation; determine the cold backup system corresponding to the stored data according to the storage environment configured on the data source end, so as to acquire the cold backup data corresponding to the stored data through the cold backup system; A transmission unit is configured to acquire network transmission parameters between the data source and the migration target, including transmission rate, network speed, and network latency; compress the cold backup data based on the network transmission parameters to obtain compressed data; segment the compressed data according to a preset segmentation granularity to obtain multiple segmented data; and transmit the segmented data from the data source to the migration target through a concurrent channel, wherein the storage cluster corresponding to the data source is different from the storage cluster corresponding to the migration target. The processing unit is used to control the migration target end to process the segmented data to obtain processed data, the processing including decompression; The processing unit is further configured to write the processed data to the migration target end according to a preset rule, including: determining the preset rule based on the storage environment configured on the data source end and the storage environment configured on the migration target end; and writing the processed data to the migration target end according to the preset rule, wherein the preset rule is determined based on the storage environment configured on the migration target end. Specifically, the transmission unit is used for: If the network transmission parameters indicate a network anomaly between the data source and the migration target, the split data is transmitted from the data source to the relay cluster. The split data is transmitted to the migration target end through the relay cluster.

6. The apparatus according to claim 5, characterized in that, The transmission unit is specifically used for: Monitor the network transmission parameters between the data source and the migration target. If the network transmission parameters meet the transmission conditions, the cold backup data is segmented according to the preset segmentation granularity to obtain multiple transmission data. The transmission data is transmitted from the data source to the migration target through the concurrent channel.

7. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store program code; the processor is used to execute the data processing method according to any one of claims 1 to 4 according to the instructions in the program code.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the data storage processing method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the steps of the data processing method according to any one of claims 1 to 4.

Images