Data migration method, device, equipment, and storage medium

Abstract

The present application discloses a method, apparatus, device, and storage medium for migrating very large data, relating to the field of cloud service technology, including: obtaining a data migration request and creating a target disk of the same size as the source disk; if new data is written to the source disk during the process of reading source data from the source disk through a data reading function, querying the sectors in the source disk used to store data to obtain the target sectors storing the new data, and saving the target sector changes to a preset file directory; querying the target sectors from the preset file directory, traversing and reading the new data in the target sectors, and compressing and writing the new data to the target disk. Abandoning the imperfect SDK interface provided by VMware, using the read function of the C library to implement reading and writing, achieving offline migration of virtual machine business data, using independent files to record data change sectors, and ensuring that data is not lost in migration scenarios where data is written online through global traversal, and re-implementing the compression migration interface to improve transmission efficiency.

Description

Technical Field

[0001] The present invention relates to the field of cloud service technology, and in particular to a data migration method, device, equipment, and storage medium. Background Art

[0002] With the growing demand for cloud deployment and cloud applications in China, major cloud vendors are rushing to develop their own products to capture the market. Each cloud vendor's data center, where they provide services, typically hosts core customer data. Therefore, cross-platform data migration is a crucial feature for their products to gain a larger market share. Proper support for cross-platform data migration from other platforms to their own platforms significantly increases the potential for replacing competing products. Consequently, cloud vendors are placing great emphasis on implementing this feature. VMware, a global leader in virtualization products, has captured a significant portion of the domestic market. However, with the rise of localization and the expansion of customer needs, domestic vendors are also experiencing significant growth opportunities. Compatibility with VMware has become an industry standard, and the ability to support cross-platform migration of customer virtual machines from VMware to their own platforms has become a key market criterion. However, VMware products have numerous version numbers, inherent compatibility issues, and incomplete external SDK interfaces. Consequently, various challenges arise during integration and migration. One of the more common problems is that the support provided by various manufacturers for the cross-platform migration of very large disks on the VMware platform is less than ideal, and R&D personnel are often required to intervene to solve various problems such as data corruption and slow migration.

[0003] In summary, how to achieve ultra-large data migration, remove the restriction that disk data larger than 2TB cannot be normally migrated, and avoid the loss of business data during online / offline migration of virtual machines are technical problems to be solved in this field. Summary of the Invention

[0004] In view of this, the present invention aims to provide a data migration method, apparatus, device, and storage medium that can achieve ultra-large data migration, remove the limitation that disk data larger than 2TB cannot be normally migrated, and avoid the loss of business data during online / offline migration of virtual machines. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a data migration method, comprising:

[0006] Get a data migration request and create a target disk with the same size as the source disk;

[0007] If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain a target sector for storing the new data, and the target sector changes are saved in a preset file directory;

[0008] The target sector is searched from the preset file directory, the new data in the target sector is traversed and read, and the new data is compressed and written into the target disk.

[0009] Optionally, after obtaining the data migration request and creating a target disk with the same size as the source disk, the method further includes:

[0010] If no new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the obtained source data is compressed and written to the target disk.

[0011] Optionally, if new data is written to the source disk during the process of reading source data from the source disk using the data reading function, the method further includes:

[0012] A snapshot process is performed on the source disk in the current state to obtain current snapshot information of the source disk.

[0013] Optionally, querying the target sector from the preset file directory, traversing and reading the new data in the target sector, and compressing and writing the new data to the target disk includes:

[0014] Determine the target sector from all sectors based on a preset read instruction and the number of sectors and traverse and read the new data;

[0015] An offset for writing data is set, and the new data is written to the target disk based on the offset.

[0016] Optionally, compressing and writing the new data to the target disk includes:

[0017] Compressing the new data according to a preset compression method to obtain compressed new data;

[0018] The compressed new data is encrypted and transmitted to the target disk.

[0019] Optionally, the process of encrypting and transmitting the compressed new data to the target disk further includes:

[0020] Calculate the migration task progress to obtain the remaining data volume and average transfer speed;

[0021] The remaining migration time of the data transmission is calculated based on the remaining data amount and the average transmission speed.

[0022] Optionally, after calculating the remaining migration time for data transmission based on the remaining data amount and the average transmission speed, the method further includes:

[0023] The data upload information including the remaining migration time, the remaining data volume and the average transmission speed is pushed to the service platform so that the data upload information is displayed when the user hovers the mouse over the data transmission task bar on the service platform.

[0024] In a second aspect, the present application discloses a data migration device, comprising:

[0025] The disk creation module is used to obtain a data migration request and create a target disk with the same size as the source disk;

[0026] A new data saving module is configured to query sectors in the source disk for storing data to obtain target sectors for storing the new data if new data is written to the source disk during the process of reading source data from the source disk through a data reading function, and save the target sector changes to a preset file directory;

[0027] The data writing module is used to query the target sector from the preset file directory, traverse and read the new data in the target sector, and compress and write the new data into the target disk.

[0028] In a third aspect, the present application discloses an electronic device, comprising:

[0029] Memory, used to store computer programs;

[0030] The processor is used to execute the computer program to implement the steps of the aforementioned disclosed data migration method.

[0031] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed data migration method are implemented.

[0032] It can be seen that the present application discloses a data migration method, including: obtaining a data migration request and creating a target disk with the same size as the source disk; if there is new data written to the source disk during the process of reading the source data from the source disk through the data reading function, then querying the sectors in the source disk for storing data to obtain the target sectors for storing the new data, and saving the target sector changes to a preset file directory; querying the target sectors from the preset file directory, traversing and reading the new data in the target sectors, and compressing and writing the new data to the target disk. It can be seen that the present application abandons the imperfect SDK interface provided by VMware, re-implements the reading function of the C library to achieve offline migration of virtual machine business data, and uses independent files to record data change sectors, and realizes data loss in online migration scenarios with data writing through global traversal, and removes the limitation on the amount of large data migration in the process of data migration using VMware, and re-implements the compression migration interface to improve transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0034] Figure 1 A flow chart of a data migration method disclosed in this application;

[0035] Figure 2 A flowchart of a specific data migration method disclosed in this application;

[0036] Figure 3 This is a flow chart of another specific data migration method disclosed in this application;

[0037] Figure 4 This is a flowchart of a specific disk migration method disclosed in this application;

[0038] Figure 5 This is a schematic diagram of the structure of a data migration device disclosed in this application;

[0039] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] With the growing demand for cloud deployment and cloud applications in China, major cloud vendors are rushing to develop their own products to capture the market. Each cloud vendor's data center, where they provide services, typically hosts core customer data. Therefore, cross-platform data migration is a crucial feature for their products to gain a larger market share. Proper support for cross-platform data migration from other platforms to their own platforms significantly increases the potential for replacing competing products. Consequently, cloud vendors are placing great emphasis on implementing this feature. VMware, a global leader in virtualization products, has captured a significant portion of the domestic market. However, with the rise of localization and the expansion of customer needs, domestic vendors are also experiencing significant growth opportunities. Compatibility with VMware has become an industry standard, and the ability to support cross-platform migration of customer virtual machines from VMware to their own platforms has become a key market criterion. However, VMware products have numerous version numbers, inherent compatibility issues, and incomplete external SDK interfaces. Consequently, various challenges arise during integration and migration. One of the more common problems is that the support provided by various manufacturers for the cross-platform migration of very large disks on the VMware platform is less than ideal, and R&D personnel are often required to intervene to solve various problems such as data corruption and slow migration.

[0042] To this end, the present application discloses a data migration solution that can realize ultra-large data migration, remove the restriction that disk data larger than 2T cannot be migrated normally, and avoid the loss of virtual machine business data during online / offline migration.

[0043] Reference Figure 1 As shown, an embodiment of the present invention discloses a data migration method, including:

[0044] Step S11: Obtain a data migration request and create a target disk with the same size as the source disk.

[0045] In this embodiment, the corresponding data migration request is first obtained, and then the oversized disk is reconstructed and migrated. It is understandable that when migrating oversized disks without reconstructing, migrating disks larger than 2TB using VMware's VDDK (Virtual Disk Development Kit) is restricted. Using VMware-provided interfaces to implement migration will result in a direct error message, such as a server-refused connection. This migration error primarily arises when migrating a virtual machine and calling VixDiskLib_Clone(), which indirectly calls the read interface VixDiskLib_Read() and the write interface VixDiskLib_Write(). Calling the write interface results in an out-of-bounds error. Therefore, in this embodiment, the virtualization product ICS reimplements the migration logic when migrating oversized data on the VMware platform. Regardless of the source disk format, ICS creates a blank disk of the same size in raw format by default. The raw format is chosen here, which is the most stable for data storage. The VixDiskLib_Write() function is replaced with the write function in the C library to directly save the read data. This logically resolves the read problem.

[0046] In this embodiment, after obtaining a data migration request and creating a target disk of the same size as the source disk, the process further includes: if no new data is written to the source disk during the process of reading source data from the source disk using a data read function, compressing and writing the obtained source data to the target disk. It is understood that when performing offline data migration, all data on the source disk is read and saved using the write function in the C library, and the data is saved in the most stable raw format.

[0047] Step S12: If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain the target sectors for storing the new data, and the target sector changes are saved in the preset file directory.

[0048] In this embodiment, if there is online data migration, that is, new data is written to the source disk while the source data is being read from the source disk using the data read function, then the read function cannot read the data correctly in one go due to the continuous data being written. Therefore, without affecting the use of the source disk, the new data is saved in the preset file directory by changing the sectors where the new data is found.

[0049] In this embodiment, if new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the process further includes: taking a snapshot of the source disk in its current state to obtain the current snapshot information of the source disk. It is understandable that, through the vSphere API, the platform takes a virtual snapshot, then calls QueryChangeDiskAreas() to query the changed sectors, and saves the changed areas of the virtual disk in / var / lib / ics / v2v_offset.data. The underlying layer traverses and reads the data based on the disk data recorded in v2v_offset.data, and writes it directly to the virtual raw format disk. The main implementation method is: first use the query function to query the changed sectors, VixDiskLib_Read(src.Handle(), offset+j*secNum, secNum, buf); where secNum is the sector number; then set the write offset, lseek(ofd, vec[i].offset, SEEK_SET); finally save the data to the raw virtual disk, write(ofd, buf, VIXDISKLIB_SECTOR_SIZE * secNum); through the above method, encapsulate it as QueryChangeDiskAreas() to traverse sector changes, optimize the implementation to remove the limit of traversing more than 2000 items at a time, implement global traversal to ensure that all sector changes can be read, and solve the data loss problem.

[0050] Step S13: querying the target sector from the preset file directory, traversing and reading the new data in the target sector, and compressing and writing the new data into the target disk.

[0051] In this embodiment, based on a preset read instruction and the number of sectors, the target sector is determined from all sectors and the new data is traversed and read; an offset for writing the data is set, and the new data is written to the target disk based on the offset. It is understandable that after the new data changes are saved to another location, the target sector is determined from all sectors based on the corresponding sector query information in the preset file directory. The target sector is the sector that stores the new data. Then, all the new data therein is compressed and written to the target disk, completing the large data migration activity. This avoids the inability to successfully read data in one go due to the continuous writing of new data during the online process, thereby eliminating the limitation of traversing more than 2000 items at a time and achieving a global traversal to ensure that the change of all sectors can be read to avoid the problem of new data loss caused by loss.

[0052] It can be seen that the present application discloses a data migration method, including: obtaining a data migration request and creating a target disk with the same size as the source disk; if there is new data written to the source disk during the process of reading the source data from the source disk through the data reading function, then querying the sectors in the source disk for storing data to obtain the target sectors for storing the new data, and saving the target sector changes to a preset file directory; querying the target sectors from the preset file directory, traversing and reading the new data in the target sectors, and compressing and writing the new data to the target disk. It can be seen that the present application abandons the imperfect SDK interface provided by VMware, re-implements the reading function of the C library to achieve offline migration of virtual machine business data, and uses independent files to record data change sectors, and realizes data loss in online migration scenarios with data writing through global traversal, and removes the limitation on the amount of large data migration in the process of data migration using VMware, and re-implements the compression migration interface to improve transmission efficiency.

[0053] Reference Figure 2 As shown, the embodiment of the present invention discloses a specific data migration method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0054] Step S21: Obtain a data migration request and create a target disk with the same size as the source disk.

[0055] Step S22: If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain the target sectors for storing the new data, and the target sector changes are saved in the preset file directory.

[0056] For more detailed processes in steps S21 and S22, please refer to the aforementioned disclosed embodiments, which will not be described again here.

[0057] Step S23: query the target sector from the preset file directory, traverse and read the new data in the target sector, compress the new data according to a preset compression method to obtain compressed new data; encrypt the compressed new data and transmit it to the target disk.

[0058] In this embodiment, the migration of ultra-large disks on the VMware side can achieve data change search and migration through the above process. However, since the disk data is more than 2T, the migration efficiency will also be a pain point. Traversing all the data blocks on the disk and the migration efficiency are also common pain points of current industry products. To solve this problem, the ICS cross-cloud migration solution implements the addition of data compression transmission to improve the transmission rate. First, the interface is encapsulated, for example: encapsulate the VixDiskLIb_Open interface, specify the compression method according to the specific business data, and open the NBDSSL transmission. The specific design is as follows: First, the business data to be migrated is compressed according to the preset compression method, and the business data is compressed in the following way: vixError=VixDiskLib_Open(appGlobals.connection,appGlobals.diskPath,appGlobals.openFlags,&srcHandle); wherein appGlobals.openFlags is used to specify the opening method, and specifies the skipz method for compression transmission. VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ opens NBDSSL transmission with skipz compression.

[0059] It can be seen that this embodiment improves transmission efficiency by re-implementing the compression migration interface. Compared with the time wasted in previous ultra-large data migration, the embodiment of the present invention can greatly improve data transmission efficiency and reduce migration time through the compression migration interface.

[0060] Reference Figure 3 As shown, the embodiment of the present invention discloses a specific data migration method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0061] Step S31: Obtain a data migration request and create a target disk with the same size as the source disk.

[0062] Step S32: If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain the target sectors for storing the new data, and the target sector changes are saved in the preset file directory.

[0063] Step S33: querying the target sector from the preset file directory, and traversing and reading the new data in the target sector.

[0064] For more detailed processes in steps S31, S32, and S33, please refer to the aforementioned disclosed embodiments, which will not be described again here.

[0065] Step S34: Calculate the migration task progress to obtain the remaining data volume and the average transmission speed; and calculate the remaining migration time of the data transmission based on the remaining data volume and the average transmission speed.

[0066] In this embodiment, refer to Figure 4 To optimize the customer experience, we've added an interface that accurately displays task progress and estimates of remaining time for large data migrations. This is because competitive products lack refined progress bar displays. Obtaining Valid Data Blocks: Based on the VDDK-provided VixDiskLib_QueryAllocatedBlocks() interface, the following API is provided for the platform to call to retrieve valid disk data. It is understandable that after data is transferred through the interface, the remaining time data and the estimated transfer speed can also be calculated in the interface, such as: vix-disklib-clone-getallocatedblocks-host xxxx....[datastore1] / dir / xxxx.vmdk". The remaining data and average transfer speed are calculated in the data transfer interface merge (interface) and pushed to the platform. The platform uses the remaining data and average transfer speed to calculate the remaining time vix-disklib-clone-merge -[datastore1] / dir / xxxx.vmdk. At the beginning of the migration, the platform obtains the total migration quantity and saves it to / var / lib / ics / v2v_remain.data in the following format: {"remaining_data":1085}. Based on the remaining data and average transfer speed calculated in the above interface, the remaining time is calculated based on the remaining data and average transfer speed.

[0067] Step S35: Pushing the data upload information including the remaining migration time, the remaining data volume and the average transmission speed to the service platform so that the data upload information is displayed when the user hovers the mouse over the data transmission task bar on the service platform.

[0068] In this embodiment, the data upload information of the remaining migration time, remaining data volume and average transmission speed obtained above is pushed to the business platform, thereby realizing the refined display of platform tasks. The specific effect is that when the mouse hovers over the task bar, the total transmission data, remaining transmission data, average transmission speed and remaining time can be displayed. Figure 4As shown in the figure, when the migration begins, the VixDiskLib_Clone() solution fails, so we restructure the read function and then use QueryChange to query the target sector. Accordingly, we save the changes using v2v_offset.data. This optimizes the original limit, which is typically 2000 entries. If more than 2000 entries are reached, an error is automatically reported and data migration is discontinued. ISC then creates a target disk of the same size in raw format. Two scenarios are considered: offline migration, where no new data is written during the migration; and online migration, where new data is continuously written. During data migration, the data to be migrated is retrieved through the Request interface, and the VixDisklb_Open compression interface is implemented to optimize transmission efficiency. After valid data is retrieved through getallcatedblocks, data estimation is performed through the merge interface, enabling a detailed display of the migration progress. At this point, the large data migration is complete.

[0069] This demonstrates that this application and reimplementation of the compression migration interface improves transmission efficiency, accurately captures valid and historical data, and calculates the remaining data volume and duration. By optimizing migration efficiency from the implementation perspective, this significantly improves migration efficiency, accurately displays task progress, and enhances the business experience. This significantly contributes to enhancing product competitiveness.

[0070] Reference Figure 5 As shown, the embodiment of the present invention discloses a specific data migration device, including:

[0071] The disk creation module 11 is used to obtain a data migration request and create a target disk with the same size as the source disk;

[0072] A new data saving module 12 is configured to, if new data is written to the source disk during the process of reading source data from the source disk through a data reading function, query the sectors in the source disk for storing data to obtain a target sector for storing the new data, and save the target sector changes to a preset file directory;

[0073] The data writing module 13 is configured to search the target sector from the preset file directory, traverse and read the new data in the target sector, and compress and write the new data into the target disk.

[0074] It can be seen that the present application discloses a data migration method, including: obtaining a data migration request and creating a target disk with the same size as the source disk; if there is new data written to the source disk during the process of reading the source data from the source disk through the data reading function, then querying the sectors in the source disk for storing data to obtain the target sectors for storing the new data, and saving the target sector changes to a preset file directory; querying the target sectors from the preset file directory, traversing and reading the new data in the target sectors, and compressing and writing the new data to the target disk. It can be seen that the present application abandons the imperfect SDK interface provided by VMware, re-implements the reading function of the C library to achieve offline migration of virtual machine business data, and uses independent files to record data change sectors, and realizes data loss in online migration scenarios with data writing through global traversal, and removes the limitation on the amount of large data migration in the process of data migration using VMware, and re-implements the compression migration interface to improve transmission efficiency.

[0075] In some specific implementations, the disk creation module 11 may specifically include:

[0076] The offline migration unit is configured to compress and write the acquired source data to the target disk if no new data is written to the source disk during the process of reading source data from the source disk through a data reading function.

[0077] In some specific implementations, the new data storage module 12 may specifically include:

[0078] The snapshot processing unit is used to perform snapshot processing on the source disk in the current state to obtain the current snapshot information of the source disk.

[0079] In some specific implementations, the data writing module 13 may specifically include:

[0080] The data traversal unit is used to determine the target sector from all sectors based on a preset read instruction and the number of sectors and traverse and read the new data; set an offset for writing data, and write the new data to the target disk based on the offset.

[0081] In some specific implementations, the data writing module 13 may specifically include:

[0082] The encryption transmission submodule is used to compress the new data according to a preset compression method to obtain compressed new data; and encrypt and transmit the compressed new data to the target disk.

[0083] In some specific implementations, the encryption transmission submodule may specifically include:

[0084] The progress calculation unit is used to calculate the progress of the migration task to obtain the remaining data volume and the average transmission speed; and calculate the remaining migration time of the data transmission based on the remaining data volume and the average transmission speed.

[0085] In some specific implementations, the data migration device may include:

[0086] The information display unit is used to push the data upload information including the remaining migration time, the remaining data volume and the average transmission speed to the service platform, so that the data upload information is displayed when the user hovers the mouse over the data transmission task bar on the service platform.

[0087] Specifically, the new data storage module 12 can be used to implement, if there is online data migration, that is, if new data is written to the source disk while the source data is being read from the source disk using the data read function, then due to the continuous data being written, the read function cannot read the data correctly in one go. Therefore, without affecting the source disk application, the file storage location of the new data is changed by saving the sectors where the new data is found to be written to the preset file directory. In this embodiment, if new data is written to the source disk while the source data is being read from the source disk using the data read function, the system further includes: taking a snapshot of the current source disk to obtain the current snapshot information of the source disk. It is understood that, through the vSphere API, the platform takes a virtual snapshot, then calls QueryChangeDiskAreas() to query the changed sectors, and saves the changed areas of the virtual disk in / var / lib / ics / v2v_offset.data. The underlying layer traverses and reads the data based on the disk data recorded in v2v_offset.data and writes it directly to the virtual raw disk. The main implementation method is as follows: first, use the query function to query the changed sectors, VixDiskLib_Read(src.Handle(), offset+j*secNum, secNum, buf); where secNum is the sector number; then set the write offset, lseek(ofd, vec[i].offset, SEEK_SET); and finally save the data to the raw virtual disk, write(ofd, buf, VIXDISKLIB_SECTOR_SIZE * secNum). Through the above method, encapsulate it as QueryChangeDiskAreas() to traverse sector changes. Optimization removes the limit of traversing more than 2000 items at a time, implements global traversal to ensure that all sector changes can be read, and solves the data loss problem.

[0088] The data writing module 13 can be specifically used to determine the target sector from all sectors based on a preset read instruction and the number of sectors and traverse and read the new data; set an offset for writing the data, and write the new data to the target disk based on the offset. It is understandable that after the new data changes are saved to another location, the target sector is determined from all sectors based on the corresponding sector query information in the preset file directory. The target sector is the sector that stores the new data, and then all the new data therein is compressed and written to the target disk to complete the large data migration activity, avoiding the problem of data being unable to be read successfully at one time due to continuous writing of new data during the online process, thereby eliminating the limitation of traversing more than 2000 items at a time, and achieving global traversal to ensure that the change of all sectors can be read to avoid the problem of new data loss caused by loss.

[0089] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0090] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the following steps:

[0091] Get a data migration request and create a target disk with the same size as the source disk;

[0092] If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain a target sector for storing the new data, and the target sector changes are saved in a preset file directory;

[0093] The target sector is searched from the preset file directory, the new data in the target sector is traversed and read, and the new data is compressed and written into the target disk.

[0094] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0095] If no new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the obtained source data is compressed and written to the target disk.

[0096] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0097] A snapshot process is performed on the source disk in the current state to obtain current snapshot information of the source disk.

[0098] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0099] Determine the target sector from all sectors based on a preset read instruction and the number of sectors and traverse and read the new data;

[0100] An offset for writing data is set, and the new data is written to the target disk based on the offset.

[0101] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0102] Compressing the new data according to a preset compression method to obtain compressed new data;

[0103] The compressed new data is encrypted and transmitted to the target disk.

[0104] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0105] Calculate the migration task progress to obtain the remaining data volume and average transfer speed;

[0106] The remaining migration time of the data transmission is calculated based on the remaining data amount and the average transmission speed.

[0107] In some specific implementations, the processor may implement the following steps by executing the computer program stored in the memory:

[0108] The data upload information including the remaining migration time, the remaining data volume and the average transmission speed is pushed to the service platform so that the data upload information is displayed when the user hovers the mouse over the data transmission task bar on the service platform.

[0109] In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0110] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0111] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0112] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, enabling the processor 21 to calculate and process the massive amount of data 223 in the memory 22. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of performing the data migration method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer programs 222 can also include computer programs capable of performing other specific tasks. The data 223 can include data received by the electronic device from external devices as well as data collected by its own input / output interface 25.

[0114] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned disclosed data migration method. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0116] Those skilled in the art will further appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules can be stored in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0117] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0118] The above is a detailed introduction to the data migration method, device, equipment, and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, based on the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data migration method, characterized in that: include: Get a data migration request and create a target disk with the same size as the source disk; If new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the sectors in the source disk used to store data are queried to obtain a target sector for storing the new data, and the target sector changes are saved in a preset file directory; Querying the target sector from the preset file directory, traversing and reading the new data in the target sector, and compressing and writing the new data to the target disk to bypass VMware restrictions when migrating data exceeding 2TB; Creating a target disk with the same size as the source disk includes: Create a blank RAW disk of the same size as the source disk, and then use the write function of the C library to save the read data to the blank RAW disk; The querying of the sectors in the source disk for storing data to obtain the target sectors for storing the new data, and saving the target sector changes to a preset file directory, includes: The changed sectors in the source disk are queried using a data change query instruction to obtain a target sector, a write offset of the write data is set, and the write data is written according to the write offset through a write function.

2. The data migration method according to claim 1, characterized in that: After obtaining the data migration request and creating a target disk with the same size as the source disk, the method further includes: If no new data is written to the source disk during the process of reading source data from the source disk through the data reading function, the obtained source data is compressed and written to the target disk.

3. The data migration method according to claim 1, wherein: If new data is written to the source disk during the process of reading source data from the source disk using the data reading function, the method further includes: A snapshot process is performed on the source disk in the current state to obtain current snapshot information of the source disk.

4. The data migration method according to claim 1, wherein: The step of searching the target sector from the preset file directory, traversing and reading the new data in the target sector, and compressing and writing the new data into the target disk includes: Determine the target sector from all sectors based on a preset read instruction and the number of sectors and traverse and read the new data; An offset for writing data is set, and the new data is written to the target disk based on the offset.

5. The data migration method according to claim 1, wherein: The step of compressing and writing the new data into the target disk comprises: Compressing the new data according to a preset compression method to obtain compressed new data; The compressed new data is encrypted and transmitted to the target disk.

6. The data migration method according to any one of claims 1 to 5, characterized in that: The process of encrypting and transmitting the compressed new data to the target disk further includes: Calculate the migration task progress to obtain the remaining data volume and average transfer speed; The remaining migration time of the data transmission is calculated based on the remaining data amount and the average transmission speed.

7. The data migration method according to claim 6, characterized in that: After calculating the remaining migration time of data transmission based on the remaining data amount and the average transmission speed, the method further includes: The data upload information including the remaining migration time, the remaining data volume and the average transmission speed is pushed to the service platform so that the data upload information is displayed when the user hovers the mouse over the data transmission task bar on the service platform.

8. A data migration device, characterized in that: include: The disk creation module is used to obtain a data migration request and create a target disk with the same size as the source disk; A new data saving module is configured to query sectors in the source disk for storing data to obtain target sectors for storing the new data if new data is written to the source disk during the process of reading source data from the source disk through a data reading function, and save the target sector changes to a preset file directory; A data writing module is used to query the target sector from the preset file directory, traverse and read the new data in the target sector, and compress and write the new data to the target disk to bypass VMware restrictions when migrating data exceeding 2TB; The disk creation module is specifically used to create a blank disk in RAW format with the same size as the source disk, and then save the read data to the blank disk in RAW format through the write function of the C library; The new data saving module is specifically used to use the data change query instruction to query the changed sector in the source disk to obtain the target sector, set the offset of the write offset of the write data, and write the write data according to the write offset through the write function.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the data migration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

Images