A data migration method, device, equipment and medium

Abstract

The application discloses a data migration method and device, equipment and medium, and relates to the technical field of information. The method comprises the following steps: acquiring a first data packet containing metadata information of a target file to be aggregated which is sent by a metadata server; analyzing the first data packet and reading the target file to be aggregated from a source object storage resource of a distributed storage system according to the metadata information of the target file to be aggregated in the first data packet; and writing the target file to be aggregated into an aggregation cache. When the size of the aggregation cache meets a preset cache size, the data in the aggregation cache is flushed to a target object storage resource, and a corresponding aggregated file is generated, so as to complete data migration. Through the technical solution, the number of underlying objects is reduced in an aggregated manner, so that the performance of data reconstruction is improved and the utilization rate of a disk is improved. In conclusion, the application can improve the system performance when data migration of a distributed storage system is performed.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a data migration method, apparatus, device, and medium. Background Technology

[0002] Ceph is a system that provides distributed storage services to external cloud platforms. Based on Ceph, it can provide object storage services, block storage services, file system storage services, and other distributed storage services. The most fundamental element of a storage system is the disk; all data is stored on disks, whether HDDs, SSDs, or NMVE disks. The storage system uses certain algorithms to group these disks into storage pools, providing secure and reliable storage services. However, many distributed storage projects involve massive scenarios with hundreds of billions of small files, with an average file size of 100KB, and some files smaller than 4KB. Each small file is stored on Ceph as an object, meaning each small file occupies one object. Billions of files result in billions of underlying storage objects across the cluster. When data reconstruction is needed due to disk failure, data recovery is performed at the object level. In scenarios with massive amounts of underlying objects, data recovery from disk failures is disastrous, with the bottleneck being the disk I / O (Input / Output) interface.

[0003] The process of reclaiming data from a failed hard drive and transferring it to a new hard drive, and then recovering the data from the failed hard drive from other working hard drives, is called data reconstruction. To reduce the number of underlying data objects and address the slow recovery of massive amounts of small file data, current distributed storage approaches involve accumulating data from multiple small files in memory to 4MB or periodically flushing them to disk in the Lib module, thus reducing the number of underlying objects. However, since the data is accumulated in ordinary memory, it can be lost in case of process anomalies or node power failures. Therefore, it cannot be used in production environments where data reliability cannot be guaranteed. In conclusion, how to improve system performance during data migration in distributed storage systems remains a challenge and requires further investigation. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data migration method, apparatus, device, and medium that can improve system performance during data migration in a distributed storage system. The specific solution is as follows:

[0005] In a first aspect, this application discloses a data migration method, including:

[0006] Obtain the first data packet containing metadata information of the target file to be aggregated, sent by the metadata server;

[0007] The first data packet is parsed, and the target file to be aggregated is read from the original object storage resources of the distributed storage system based on the metadata information of the target file to be aggregated in the first data packet.

[0008] The target file to be aggregated is written into the aggregation cache. When the size of the aggregation cache meets the preset cache size, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration.

[0009] Optionally, before obtaining the first data packet containing metadata information of the target file to be aggregated sent by the metadata server, the method further includes:

[0010] Retrieve filename aggregation conditions, global aggregation conditions, and directory aggregation conditions configured through preset interfaces in the management software.

[0011] Optionally, before obtaining the first data packet containing metadata information of the target file to be aggregated sent by the metadata server, the method further includes:

[0012] The target business request is obtained through the metadata server, and the target file that meets the file name aggregation condition is selected from the business files corresponding to the target business request. Then, the metadata information of the target file is added to the preset index table.

[0013] The metadata server scans the preset index table according to a preset period and filters out the target files that meet the global aggregation conditions or the directory aggregation conditions. Then, the target files to be aggregated are added to the aggregateable list and a first data packet containing the metadata information of the target files to be aggregated is sent.

[0014] Optionally, after obtaining the target business request from the metadata server and filtering out the target files that meet the filename aggregation conditions from the business files corresponding to the target business request, and then adding the metadata information of the target files to the preset index table, the method further includes:

[0015] The metadata server scans the preset index table according to a preset period and filters out non-aggregated files in the target files that do not meet the global aggregation conditions and the directory aggregation conditions.

[0016] The non-aggregated files are added to the non-aggregation list by the metadata server, and a second data packet containing the metadata information of the non-aggregated files is sent.

[0017] Optionally, after adding the non-aggregated files to the non-aggregation list via the metadata server and sending a second data packet containing metadata information of the non-aggregated files, the method further includes:

[0018] Obtain the second data packet sent by the metadata server, which contains metadata information of the non-files to be aggregated;

[0019] The second data packet is parsed and the non-aggregated file is read from the original object storage resource of the distributed storage system according to the metadata information of the non-aggregated file in the second data packet. Then the non-aggregated file is saved to the target object storage resource.

[0020] Optionally, writing the target file to be aggregated into the aggregation cache includes:

[0021] Based on the metadata information of the target file to be aggregated, the global aggregation conditions, and the directory aggregation conditions, the target aggregation conditions corresponding to the target file to be aggregated are determined, and the target file to be aggregated is written into the aggregation cache according to the aggregation strategy corresponding to the target aggregation conditions.

[0022] Optional, also includes:

[0023] Obtain the file write request for the file to be written in the aggregate file and determine the file write type;

[0024] If the file write type is a modification write type within the size range of the aggregate file, then the file is directly written to the aggregate file and flushed to the target object storage resource;

[0025] If the file write type is an append write type that exceeds the size range of the aggregated file, then the file to be written is read from the aggregated file into the buffer and appended to obtain the written file. Then, the written file is flushed to the target object storage resource and the corresponding metadata information is updated.

[0026] Secondly, this application discloses a data migration apparatus, comprising:

[0027] The data packet acquisition module is used to acquire the first data packet containing metadata information of the target file to be aggregated, sent by the metadata server;

[0028] The file to be aggregated reading module is used to parse the first data packet and read the target file to be aggregated from the original object storage resources of the distributed storage system according to the metadata information of the target file to be aggregated in the first data packet;

[0029] The file migration module is used to write the target file to be aggregated into the aggregation cache. When the size of the aggregation cache meets the preset cache size, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration.

[0030] Thirdly, this application discloses an electronic device, including:

[0031] Memory, used to store computer programs;

[0032] A processor is configured to execute the computer program to implement the steps of the data migration method disclosed above.

[0033] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned data migration method.

[0034] In this application, during data migration, a first data packet containing metadata information of the target file to be aggregated, sent by a metadata server, is first obtained. This first data packet is then parsed, and the target file to be aggregated is read from the original object storage resource of the distributed storage system based on the metadata information of the target file to be aggregated within the first data packet. The target file to be aggregated is then written into an aggregation cache. When the size of the aggregation cache meets a preset cache size, the data in the aggregation cache is flushed to the target object storage resource, and a corresponding aggregation file is generated to complete the data migration. Therefore, in this application, during data migration in a distributed storage system, the first step is to obtain a data packet containing metadata information of the target file to be aggregated from a metadata server. The target file to be aggregated is then located in the original object storage resource using the metadata information of the target file to be aggregated, and written into an aggregation cache. When the aggregation cache meets a preset cache size, the data in the aggregation cache is flushed to the target object storage resource to complete the data migration. Therefore, when migrating data in a distributed storage system, this application reads the target file to be aggregated from the original object storage resource in the background, writes the target file to be aggregated to the aggregation cache, and then writes the data in the aggregation cache to the target object storage resource. This aggregation method reduces the number of underlying objects, thereby improving data reconstruction performance and disk utilization. Furthermore, when the aggregation cache reaches a preset cache size, the data is flushed down, avoiding data accumulation in ordinary memory and preventing data loss in case of process abnormalities or node power failures. In summary, this application can improve system performance during data migration in a distributed storage system. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 A flowchart of a data migration method provided in this application;

[0037] Figure 2 A flowchart of a specific data migration method provided in this application;

[0038] Figure 3 A schematic diagram of the data migration process provided for this application;

[0039] Figure 4 A flowchart of a specific data migration method provided in this application;

[0040] Figure 5 A schematic diagram of a data migration device provided in this application;

[0041] Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] To reduce the number of underlying data objects and address the slow recovery of massive amounts of small file data, existing distributed storage approaches involve accumulating data from multiple small files in memory to a maximum of 4MB or periodically flushing them to disk within the Lib module. This reduces the number of underlying objects. However, since the data is accumulated in ordinary memory, it is susceptible to loss in cases of process anomalies or node power failures. Therefore, it is unusable in live networks where data reliability cannot be guaranteed. To address this, this application provides a data migration method that improves system performance during data migration in distributed storage systems.

[0044] This invention discloses a data migration method, see [link to relevant documentation]. Figure 1 As shown, the method includes:

[0045] Step S11: Obtain the first data packet containing metadata information of the target file to be aggregated, sent by the metadata server.

[0046] In this embodiment, the metadata server (MDS) is a server that organizes, stores, and manages data metadata according to a data structure. Specifically, the metadata server stores the relevant metadata of the data in the original object storage resources of the distributed storage system. It can be understood that the metadata server first packages the metadata information of the target file to be aggregated and sends it to the backend. The backend receives the first data packet containing the metadata information of the target file to be aggregated. Through this technical solution, the backend obtains the metadata information of the target file to be aggregated, enabling it to subsequently read the target file to be aggregated from the original object storage resources of the distributed storage system using this metadata information, and further perform relevant migration operations on the target file to be aggregated.

[0047] Step S12: Parse the first data packet and read the target file to be aggregated from the original object storage resources of the distributed storage system according to the metadata information of the target file to be aggregated in the first data packet.

[0048] In this embodiment, the first data packet is a data packet containing metadata information of the target file to be aggregated. The target file to be aggregated is read from the original object storage resource of the distributed storage system by parsing the first data packet and based on the information in the first data packet. It is understood that the original object storage resource (i.e., ObjectStorage Device, OSD) is the object storage resource originally used to store related files in the distributed storage system. The target file to be aggregated is read from the original object storage resource of the distributed storage system. Through the above technical solution, the target file to be aggregated is read from the original object storage resource of the distributed storage system based on the metadata information of the target file to be aggregated, so that the target file to be aggregated can be aggregated and then saved to the target object storage resource.

[0049] Step S13: Write the target file to be aggregated into the aggregation cache. When the size of the aggregation cache meets the preset cache size, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration.

[0050] In this embodiment, the preset cache size is obtained in advance according to the corresponding preset interface. Users can set the preset cache size themselves through the corresponding preset interface according to actual needs. In one specific implementation, the preset cache size is 4M, that is, the target file to be aggregated is written into the aggregation cache. When the aggregation cache size is full of 4M, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration; conversely, when the aggregation cache size is less than 4M, the next file is processed until all the data in the first data packet is processed. Through the above technical solution, the target file to be aggregated is aggregated and then flushed to the target object storage resource to complete the data migration. The aggregation method reduces the number of underlying objects, thereby improving the performance of data reconstruction and increasing disk utilization. At the same time, it avoids the problem of data loss caused by data accumulation on ordinary memory in abnormal situations such as process abnormalities or node power failures.

[0051] As can be seen, in this embodiment, when migrating data in a distributed storage system, a data packet containing metadata information of the target file to be aggregated is first obtained from the metadata server. The target file to be aggregated is then located in the original object storage resource using this metadata information and written into the aggregation cache. When the aggregation cache reaches a preset cache size, the data in the aggregation cache is flushed to the target object storage resource to complete the data migration. Therefore, when migrating data in a distributed storage system, this application reads the target file to be aggregated from the original object storage resource in the background, writes it to the aggregation cache, and then writes the data in the aggregation cache to the target object storage resource. This aggregation method reduces the number of underlying objects, thereby improving data reconstruction performance and disk utilization. Furthermore, writing data to the aggregation cache and flushing it when the aggregation cache reaches a preset cache size avoids data accumulation in ordinary memory, which could lead to data loss in case of process abnormalities or node power failures. In summary, this application can improve system performance during data migration in a distributed storage system.

[0052] See Figure 2 As shown, this embodiment of the invention discloses a specific file writing method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0053] Step S21: Obtain the filename aggregation conditions, global aggregation conditions, and directory aggregation conditions configured through the preset interface in the management software.

[0054] In this embodiment, before obtaining the filename aggregation conditions, global aggregation conditions, and directory aggregation conditions configured through a preset interface in the management software, the process further includes: configuring filename aggregation conditions, global aggregation conditions, and directory aggregation conditions through a preset interface in the management software, and connecting to the underlying Ceph interface. Specifically, the filename aggregation conditions are pre-set files that meet the corresponding filename conditions, allowing for the filtering of subsequently obtained business files based on these conditions, using the business files that meet the filename aggregation conditions as target files. The global aggregation conditions aim to aggregate small files scanned by the backend into one or more large files and write them to the same storage pool if users do not frequently access small files. The directory aggregation conditions aim to aggregate small files in the same directory into one or more large files and write them to the same storage pool if users do not frequently access small files in that directory. It should be noted that the aggregation effect is relatively poor if the number of small files in the directory is small. The above technical solution configures and obtains filename aggregation conditions, global aggregation conditions, and directory aggregation conditions, so as to facilitate subsequent aggregation and filtering of business files corresponding to the target business request, and aggregation of target files according to the corresponding aggregation type, thereby improving the efficiency of file migration and the performance of data reconstruction.

[0055] Step S22: Obtain the target business request through the metadata server and filter out the target file that meets the filename aggregation condition from the business file corresponding to the target business request. Then add the metadata information of the target file to the preset index table.

[0056] In this embodiment, the metadata server obtains a target business request, filters out target files that meet the filename aggregation conditions from the business files corresponding to the target business request, and adds the metadata information of the target files to a preset index table. Specifically, when the metadata server obtains the aggregation conditions set by the management software, saves the obtained aggregation conditions, and then calls the create function to create an index table, according to Map...<key,vec> Data is stored in a specific manner. When a target business request is received, including but not limited to open, rename, and setxattr, a response interruption is triggered. Business files are then processed in a loop based on aggregation conditions to filter out the corresponding target files. This technical solution filters the business files corresponding to the target business request based on filename aggregation conditions, selecting the target files that meet the criteria. This accelerates file migration processing and improves data reconstruction performance.

[0057] Step S23: The metadata server scans the preset index table according to a preset period and filters out the target files in the target files that meet the global aggregation conditions or the directory aggregation conditions. Then, the target files to be aggregated are added to the aggregateable list and a first data packet containing the metadata information of the target files to be aggregated is sent.

[0058] In this embodiment, the metadata server scans and filters target files in the target files that meet the aggregation conditions according to a preset period, adds the target files to the aggregateable list, and sends a data packet containing the metadata information of the target files. Specifically, the metadata server periodically scans the index table according to the aggregation conditions to determine whether a file meets the hierarchical aggregation migration conditions, adds the target files to the aggregateable list, creates a task, packages it, and sends it to the background for processing. After the hierarchical aggregation operation is completed, the record is deleted from the index table. Through the above technical solution, the target files to be aggregated in the target files are filtered out by the global aggregation conditions and the directory aggregation conditions, so as to facilitate the subsequent aggregation and migration of the target files.

[0059] Step S24: Obtain the first data packet containing metadata information of the target file to be aggregated, sent by the metadata server.

[0060] Step S25: Parse the first data packet and read the target file to be aggregated from the original object storage resource of the distributed storage system according to the metadata information of the target file to be aggregated in the first data packet.

[0061] In this embodiment, the first data packet is parsed, and the target file to be aggregated is read from the original object storage resource of the distributed storage system based on the metadata information of the target file to be aggregated in the first data packet. Specifically, the background obtains the migration task from the metadata server and reads the corresponding target file to be aggregated from the original object storage resource. Through the above technical solution, the target file to be aggregated is read from the original object storage resource of the distributed storage system based on the metadata information of the target file to be aggregated, so that the target file to be aggregated can be aggregated and then saved to the target object storage resource.

[0062] Step S26: Write the target file to be aggregated into the aggregation cache. When the size of the aggregation cache meets the preset cache size, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration.

[0063] In this embodiment, the target aggregation condition corresponding to the target file to be aggregated is determined based on the metadata information of the target file to be aggregated, the global aggregation condition, and the directory aggregation condition. The target file to be aggregated is then written into the aggregation cache according to the aggregation strategy corresponding to the target aggregation condition. Specifically, the background process parses the data packets sent by the metadata server, determines whether the aggregation condition is a global aggregation condition or a directory aggregation condition based on the parsed information, and records the size and index number of the received file. Then, it iterates through all files in the data packet, writing the read data into the aggregation cache according to the strategy. When the cache is full (4MB), the data is flushed to the target object storage resource. If it is less than 4MB, the next file is processed until all files in the data packet are processed. After the data in the data packet is processed, the background process calls `setxattr` to update the metadata information of the smaller files to the metadata server. The metadata server returns a success or failure message, and then the background process exits. In one specific implementation, the `get_agg_obj` function is called to retrieve the aggregate object `agg_obj` where the aggregate cache is located. The `get_agg_obj` function writes the read data to the aggregate cache based on the aggregation strategy type, the storage pool, and the index number and directory index number of the first small file in the task. When the cache is full (4MB) or all files have been processed, `aggregate_flush_obj` is called to flush the data along with the index number in the object header to the target object's storage resource. `setxattr` is then called to update the small file's metadata information: `aggregate_ino / offset / poolid / mtime`. The metadata server checks whether the creation time in the message matches the creation time in the corresponding metadata on the metadata server. If they match, the relevant metadata information is updated; otherwise, the update fails, and the failed index number information is recycled and cleared.

[0064] In this embodiment, the process further includes: obtaining a file write request for a file to be written in the aggregate file and determining the file write type; if the file write type is a modification write type within the size range of the aggregate file, then the file is directly written to the aggregate file and flushed to the target object storage resource; if the file write type is an append write type exceeding the size range of the aggregate file, then the file to be written is read from the aggregate file into a buffer and appended to obtain the written file, and then the written file is flushed to the target object storage resource and the corresponding metadata information is updated. Specifically, through Lib adaptation processing, when the front-end business performs a modification write within the file size range, the modification write is directly performed at the original position of the aggregate file, and then flushed to the object storage resource. When the front-end business performs data writing exceeding the file size, it is necessary to read the aggregated small file data and merge it with the write buffer, and then flush it to the object storage resource, changing it from an aggregated object to a regular object, while the metadata server updates the file's metadata information. In this embodiment, the data migration processing flow diagram is as follows: Figure 3 As shown, commands are configured in the management software interface; the metadata server scans the index table and matches the conditions for file migration, then packages the migration task; the background receives and parses the migration task, and simultaneously reads and writes the target file to be migrated; through Lib adaptation processing, it receives front-end business and performs file writing operations.

[0065] As can be seen, in this embodiment, by hierarchically aggregating the target files, the number of underlying objects is reduced, thereby improving data reconstruction performance and increasing disk utilization.

[0066] See Figure 4 As shown, this embodiment of the invention discloses a specific file writing method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0067] Step S31: Obtain the filename aggregation conditions, global aggregation conditions, and directory aggregation conditions configured through the preset interface in the management software.

[0068] Step S32: Obtain the target business request through the metadata server and filter out the target file that meets the filename aggregation condition from the business file corresponding to the target business request. Then, add the metadata information of the target file to the preset index table.

[0069] Step S33: The metadata server scans the preset index table according to a preset period and filters out the non-aggregated files in the target file that do not meet the global aggregation conditions and the directory aggregation conditions. Then, the non-aggregated files are added to the non-aggregation list, and a second data packet containing the metadata information of the non-aggregated files is sent.

[0070] In this embodiment, the metadata server scans the preset index table according to a preset period and filters out files in the target files that do not meet the global aggregation conditions and the directory aggregation conditions. These non-aggregated files are those that do not require aggregation and can be directly migrated. The server adds these non-aggregated files to a non-aggregation list and sends a second data packet containing their metadata information. This technical solution facilitates subsequent file migration of these non-aggregated files using their metadata information.

[0071] Step S34: Obtain the second data packet containing metadata information of the non-aggregated files sent by the metadata server.

[0072] Step S35: Parse the second data packet and read the non-aggregated file from the original object storage resource of the distributed storage system according to the metadata information of the non-aggregated file in the second data packet, and then save the non-aggregated file to the target object storage resource.

[0073] In this embodiment, the second data packet parses and reads the metadata information of the non-aggregated file in the second data packet, reads the non-aggregated file from the original object storage resource of the distributed storage system through the metadata information of the non-aggregated file, and directly saves the non-aggregated file to the target object storage to complete the data migration of the non-aggregated file.

[0074] See Figure 5 The present application discloses a data migration apparatus, comprising:

[0075] The data packet acquisition module 11 is used to acquire the first data packet containing the metadata information of the target file to be aggregated, sent by the metadata server;

[0076] The file to be aggregated reading module 12 is used to parse the first data packet and read the target file to be aggregated from the original object storage resources of the distributed storage system according to the metadata information of the target file to be aggregated in the first data packet;

[0077] The file migration module 13 is used to write the target file to be aggregated into the aggregation cache. When the size of the aggregation cache meets the preset cache size, the data in the aggregation cache is flushed to the target object storage resource and the corresponding aggregation file is generated to complete the data migration.

[0078] As can be seen, in this embodiment, when migrating data in a distributed storage system, a data packet containing metadata information of the target file to be aggregated is first obtained from the metadata server. The target file to be aggregated is then located in the original object storage resource using this metadata information and written into the aggregation cache. When the aggregation cache reaches a preset cache size, the data in the aggregation cache is flushed to the target object storage resource to complete the data migration. Therefore, when migrating data in a distributed storage system, this application reads the target file to be aggregated from the original object storage resource in the background, writes it to the aggregation cache, and then writes the data in the aggregation cache to the target object storage resource. This aggregation method reduces the number of underlying objects, thereby improving data reconstruction performance and disk utilization. Furthermore, writing data to the aggregation cache and flushing it when the aggregation cache reaches a preset cache size avoids data accumulation in ordinary memory, which could lead to data loss in case of process abnormalities or node power failures. In summary, this application can improve system performance during data migration in a distributed storage system.

[0079] In some specific embodiments, the data migration apparatus further includes:

[0080] The aggregation condition acquisition module is used to acquire filename aggregation conditions, global aggregation conditions, and directory aggregation conditions configured through preset interfaces in the management software.

[0081] In some specific embodiments, the data migration apparatus further includes:

[0082] The target file filtering module is used to obtain the target business request through the metadata server and filter out the target files that meet the file name aggregation conditions from the business files corresponding to the target business request, and then add the metadata information of the target files to the preset index table;

[0083] The module for sending files to be aggregated is used to scan the preset index table through the metadata server according to a preset period and filter out the target files in the target files that meet the global aggregation conditions or the directory aggregation conditions. Then, the target files to be aggregated are added to the aggregateable list and a first data packet containing the metadata information of the target files to be aggregated is sent.

[0084] In some specific embodiments, the data migration apparatus further includes:

[0085] The non-aggregation file filtering module is used to scan the preset index table through the metadata server according to a preset period and filter out non-aggregation files in the target file that do not meet the global aggregation conditions and the directory aggregation conditions;

[0086] The non-aggregated file packet sending module is used to add the non-aggregated files to the non-aggregation list through the metadata server and send a second data packet containing the metadata information of the non-aggregated files.

[0087] In some specific embodiments, the data migration apparatus further includes:

[0088] The non-aggregated file packet acquisition module is used to acquire the second data packet sent by the metadata server, which contains metadata information of the non-aggregated files.

[0089] The non-aggregated file migration module is used to parse the second data packet and read the non-aggregated file from the original object storage resource of the distributed storage system according to the metadata information of the non-aggregated file in the second data packet, and then save the non-aggregated file to the target object storage resource.

[0090] In some specific embodiments, the file migration module 13 is specifically used to: determine the target aggregation condition corresponding to the target file to be aggregated based on the metadata information of the target file to be aggregated, the global aggregation condition and the directory aggregation condition, and write the target file to be aggregated into the aggregation cache according to the aggregation strategy corresponding to the target aggregation condition.

[0091] In some specific embodiments, the data migration apparatus further includes:

[0092] The write request acquisition module is used to acquire file write requests for the files to be written in the aggregate file and determine the file write type.

[0093] The file modification and write module is used to directly write to the aggregate file and flush it to the target object storage resource if the file write type is a modification and write type within the size range of the aggregate file;

[0094] The file append write module is used to read the file to be written from the aggregate file into a buffer and append it to obtain the written file if the file write type is an append write type that exceeds the size range of the aggregate file. Then, the written file is flushed to the target object storage resource and the corresponding metadata information is updated.

[0095] Figure 6The illustration shows an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may further include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data migration method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0096] In this embodiment, the power supply 23 is used to provide voltage to the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to the specific application needs, and is not specifically limited here.

[0097] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0098] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20, and the computer program 222 may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data migration method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0099] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed data migration method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0100] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The above provides a detailed description of the data migration method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data migration method, characterized by, The method comprises the following steps: acquiring a first data packet containing metadata information of a target file to be aggregated sent by a metadata server; parsing the first data packet and reading the target file to be aggregated from an original object storage resource of a distributed storage system according to the metadata information of the target file to be aggregated in the first data packet; writing the target file to be aggregated into an aggregation cache, and when the size of the aggregation cache meets a preset cache size, flushing the data in the aggregation cache to a target object storage resource and generating a corresponding aggregated file to complete data migration; wherein, before acquiring the first data packet containing the metadata information of the target file to be aggregated sent by the metadata server, the method further comprises the following steps: acquiring a file name aggregation condition, a global aggregation condition and a directory aggregation condition configured through a preset interface in management software; acquiring a target service request through the metadata server and screening a target file meeting the file name aggregation condition from a service file corresponding to the target service request, and then adding the metadata information of the target file into a preset index table; scanning the preset index table according to a preset period through the metadata server and screening a target file to be aggregated meeting the global aggregation condition or the directory aggregation condition from the target file, and then adding the target file to be aggregated into an aggregable list and sending a first data packet containing the metadata information of the target file to be aggregated; and when the file completes the aggregation operation, deleting the record from the index table; the step of writing the target file to be aggregated into the aggregation cache comprises the following steps: judging a target aggregation condition corresponding to the target file to be aggregated according to the metadata information of the target file to be aggregated, the global aggregation condition and the directory aggregation condition, and writing the target file to be aggregated into the aggregation cache according to an aggregation strategy corresponding to the target aggregation condition.

2. The data migration method of claim 1, wherein, After the step of acquiring a target service request through the metadata server and screening a target file meeting the file name aggregation condition from a service file corresponding to the target service request, and then adding the metadata information of the target file into a preset index table, the method further comprises the following steps: scanning the preset index table according to a preset period through the metadata server and screening a non-aggregated file not meeting the global aggregation condition and the directory aggregation condition from the target file; adding the non-aggregated file into a non-aggregated list through the metadata server and sending a second data packet containing the metadata information of the non-aggregated file.

3. The data migration method of claim 2, wherein, After the step of adding the non-aggregated file into the non-aggregated list through the metadata server and sending a second data packet containing the metadata information of the non-aggregated file, the method further comprises the following steps: acquiring a second data packet containing the metadata information of the non-aggregated file sent by the metadata server; parsing the second data packet and reading the non-aggregated file from an original object storage resource of a distributed storage system according to the metadata information of the non-aggregated file in the second data packet, and then saving the non-aggregated file into a target object storage resource.

4. The data migration method of any of claims 1 to 3, wherein, The method further comprises the following steps: acquiring a file write request for a file to be written in the aggregate file and determining a file write type; if the file write type is a modification write type within a size range of the aggregate file, directly writing in the aggregate file and flushing to the target object storage resource; if the file write type is an append write type beyond the size range of the aggregate file, reading the file to be written from the aggregate file to a buffer, then performing append writing to obtain a written file, and then flushing the written file to the target object storage resource and updating corresponding metadata information.

5. A data migration apparatus, characterized by comprising: Comprise: a data packet acquisition module configured to acquire a first data packet containing metadata information of a target file to be aggregated sent by a metadata server; a file to be aggregated reading module configured to parse the first data packet and read the target file to be aggregated from an original object storage resource of a distributed storage system according to the metadata information of the target file to be aggregated in the first data packet; a file migration module configured to write the target file to be aggregated into an aggregate cache, and when the size of the aggregate cache meets a preset cache size, flush the data in the aggregate cache to a target object storage resource and generate a corresponding aggregate file to complete data migration; The data migration device further comprises: an aggregate condition acquisition module configured to acquire a file name aggregation condition, a global aggregation condition and a directory aggregation condition configured through a preset interface in management software; a target file screening module configured to acquire a target service request through a metadata server and screen a target file meeting the file name aggregation condition from a service file corresponding to the target service request, and then add metadata information of the target file to a preset index table; a file to be aggregated packet sending module configured to scan the preset index table according to a preset period through a metadata server, screen a target file to be aggregated meeting the global aggregation condition or the directory aggregation condition from the target file, then add the target file to be aggregated to an aggregable list, and send a first data packet containing metadata information of the target file to be aggregated; The data migration device is further configured to delete a record from the index table when a file completes an aggregation operation. The target file to be aggregated is written into the aggregate cache, comprising: determining a target aggregation condition corresponding to the target file to be aggregated according to the metadata information of the target file to be aggregated, the global aggregation condition and the directory aggregation condition, and writing the target file to be aggregated into the aggregate cache according to an aggregation strategy corresponding to the target aggregation condition.

6. An electronic device, comprising: Comprise: a memory configured to save a computer program; a processor configured to execute the computer program to implement the steps of the data migration method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the data migration method according to any one of claims 1 to 4.

Images