A method and system for migrating a storage system
By deploying a new storage system on the original storage system, reading and reconstructing data description information, and using erasure coding technology, the problems of data migration in the existing technology are solved, which is long-term, low-efficiency and high-storage space consumption, and high-efficiency is achieved. data migration and cost reduction.
Patent Information
- Application Number
- CN202011507409.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-18
AI Technical Summary
Existing storage systems consume time, low efficiency and high storage space during data migration, resulting in high migration costs.
Deploy a new storage system on the original storage system, and reconstruct the data description information by reading and importing file description information, realizing seamless migration, avoiding data copying and format conversion, and using erasure coding technology to generate verification data.
It significantly improves data migration efficiency, reduces storage space consumption and migration costs, and realizes a seamless data migration process.
Smart Images

Figure CN112558881B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data migration, and in particular to a method and system for migrating a storage system. Background Art
[0002] During the use of the storage system, the old storage system cannot meet the new requirements and needs to be upgraded. The original stored data needs to be migrated. The usual practice is to copy the data in the original storage system to the new storage system. In this process, the data copying process is very time-consuming and inefficient. At the same time, the consumption of storage space is relatively serious. During the online data migration process, twice the storage space is required at the same time.
[0003] Therefore, how to improve the efficiency of online data migration, reduce storage consumption during data migration, and reduce migration costs are issues that need to be urgently addressed in this field. Summary of the invention
[0004] The purpose of the present invention is to overcome the above-mentioned defects in the prior art and provide a method for migrating a storage system to solve the defects of the prior art in terms of high time consumption, low efficiency and high storage consumption.
[0005] To achieve the above-mentioned purpose, the first aspect of the present invention discloses a method for migrating a storage system, characterized in that the method specifically includes the following implementation steps: S1: deploying a new storage system on the original storage system; S2: reading the original file description information from the original storage system through the new storage system; S3: importing the original file description information into the new storage system; S4: reconstructing new data description information through the new storage system according to the imported original file description information; S5: putting the new storage system online to provide data services to the outside world.
[0006] Furthermore, the original file description information includes the start address and end address of the file in the actual physical storage medium.
[0007] Furthermore, the new data description information includes a node identification number, a storage device identification number, a start address and an end address.
[0008] Furthermore, it also includes migrating non-erasure coded data blocks in the original storage system into erasure coded data blocks.
[0009] Furthermore, the method also includes generating verification data according to the erasure code data block.
[0010] The second aspect of the present invention discloses a system for migrating a storage system, specifically comprising: a deployment module: used to deploy a new storage system on the original storage system; an information reading module: used to read out original file description information from the original storage system; an information import module: used to import the original file description information into the new storage system; an information reconstruction module: used to reconstruct new data description information based on the imported original file description information; a data service module: used to provide data services to the outside.
[0011] Furthermore, it also includes a data format conversion module: used to convert non-erasure code data description information in the original storage system into erasure code data description information.
[0012] Furthermore, it also includes a check code generation module: used to generate check data according to the erasure code data block, or generate check data according to the original storage system.
[0013] The third aspect of the present invention further discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store an executable program, and the processor is used to execute the executable program to implement any one of the methods for migrating a storage system described in the first aspect of the present invention.
[0014] In summary, by utilizing the method and system disclosed in the present invention, during the data migration process of the original storage system, there is no need to copy data or perform other transcoding or migration work. Instead, a new storage system is deployed on the existing storage system server to achieve seamless migration of the original data. Compared with the prior art, the problems of the data copy process being time-consuming, inefficient, and consuming serious storage space in the prior art are eliminated, the migration efficiency is greatly improved, and the data migration cost is significantly reduced. Therefore, it has obvious technical advantages and beneficial effects.
[0015] In practical applications, the modules described in the method and system disclosed in the present invention can be deployed in multiple modules on one server, or each module can be independently deployed on a different server. In particular, in order to provide more powerful computing processing capabilities, the modules can also be deployed on cluster servers as needed.
[0016] In order to have a clearer and more comprehensive understanding of the present invention, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.
[0018] Figure 1 A schematic diagram showing the main steps of a data migration method in an embodiment is shown.
[0019] Figure 2 A schematic diagram showing the principle of data migration in an embodiment is shown.
[0020] Figure 3 A schematic diagram of constructing data description in a data migration method in one embodiment is shown;
[0021] Figure 4 A schematic diagram of a data migration system in one embodiment is shown. DETAILED DESCRIPTION
[0022] See also Figure 1 , Figure 1 A schematic diagram of the main steps of a data migration method in an embodiment is shown, which specifically includes steps S11 to S13:
[0023] S11: Deploy a new storage system on the original storage system;
[0024] By deploying a new storage system to manage the physical media of servers distributed in different locations, it is independent of the physical storage media of the original storage system. By establishing a mapping relationship with the physical address of the data, operations on the data of the physical media on the original storage system (such as adding, deleting, modifying and checking) can be implemented.
[0025] S12: reading original file description information from the original storage system through the new storage system;
[0026] The new storage system must first obtain the file description information from the original storage system. For example, some storage media data is organized in the form of H1+D1 / H2+D2 / H3+D3, and the file description format is ip1:device1 / block2; other storage media data is organized in the form of H1 / D1 / D2 / D3, and the file description format may be Node1:device1 / BlockId / size.
[0027] S13: Importing the original file description information into the new storage system;
[0028] The new storage system reads the original file description information and parses it to obtain the information of the actual physical storage location of the data (such as the begin address and the end address).
[0029] S14: Reconstructing new data description information according to the imported original file description information through the new storage system.
[0030] The new storage system is completely re-established to generate new data description information based on the imported file description information. For example, the new description information is: node:device:begin:end.
[0031] S15: Putting the new storage system online to provide data operation services.
[0032] When a new storage system is launched, it can provide external data operation services (such as adding, deleting, modifying and checking).
[0033] As an example, Figure 2 As shown, Figure 2 A schematic diagram of the principle of data migration in an embodiment is shown. In this embodiment, it is assumed that the original storage system includes storage media from four different manufacturers, and the numbers of the different manufacturers are: 201, 202, 203, 204, wherein the storage media from different manufacturers are numbered a, b, c, d, and the original file system is numbered: A, B, C, D.
[0034] The principle of data migration is to deploy a new storage system in the original storage system. The new storage system does not need to modify the file system of the original storage system, nor does it require any format conversion, data copying or interaction with the original storage system. It also does not change the storage data format of the original storage medium. It only reads the file description information on the existing physical storage medium, parses it to obtain the storage location of the data, and then further reconstructs the new data description information, thereby establishing a new data management form.
[0035] For details on how to obtain new data description information from the file description information on the original storage medium, please refer to Figure 3 Schematic diagram shown.
[0036] See also Figure 3 , Figure 3 A schematic diagram of constructing data description in a data migration method in an embodiment is shown. In this embodiment, a picture file 1.jpg with a size of 10M is taken as an example. Different storage media are used for storage. For the convenience of description, they are marked as a first file 301 and a second file 302 respectively. The data organization forms and file description information of 1.jpg in different storage media are as follows:
[0037] First file 301: data organization form is: H1+D1 / H2+D2 / H3+D3; the space allocated to each storage block is the same, and when the file is stored in blocks on the physical media of D1, D2, and D3, the description information of each data block is: ip1:device1 / block2;
[0038] The second file 301: data is organized in the form of H / D5 / D6 / D7. The space allocated to each storage block is different. When the file is stored in blocks on the physical media of D5, D6, and D7, the description information of each data block is: Node1:device1 / Block / size.
[0039] The new storage system reads the description information of the above two files, parses the node number and storage medium number of each data block, and either calculates or reads the starting position and ending position of the data storage, and then generates new data description information according to the format. As an example, the new data description information includes the node identification number, the storage device identification number, the starting address and the ending address, and the format is: node: device: begin: end. Through such conversion, the new storage system obtains the new data description information of all files on the storage medium, and can further realize the takeover of the storage medium based on this.
[0040] See also Figure 4 , Figure 4 A schematic diagram of a data migration system in an embodiment is shown. In this embodiment, the data migration system includes a deployment module 401, an information reading module 402, an information importing module 403, an information reconstruction module 404, a verification code generating module 405, and a data service module 406, wherein:
[0041] Deployment module 401: used to deploy a new storage system on the original storage system;
[0042] Information reading module 402: used to read original file description information from the original storage system;
[0043] Information import module 403: used to import the original file description information into the new storage system;
[0044] Information reconstruction module 404: used to reconstruct new data description information according to the imported original file description information;
[0045] The check code generation module 405 is used to generate check data according to the erasure code data block, or to generate check data according to the original storage system;
[0046] Data service module 406: used to provide external data services;
[0047] It should be noted that, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0048] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for migrating a storage system, characterized in that: The method specifically comprises the following implementation steps: S1: Deploy a new storage system on the original storage system; S2: reading out original file description information including the start address and end address of the file in the actual physical storage medium from the original storage system through the new storage system; S3: Importing the original file description information into the new storage system; S4: reconstructing new data description information through the new storage system according to the imported original file description information, including a node identification number, a storage device identification number, a start address and an end address, in a format of node:device:begin:end; S5: Put the new storage system online to provide data services externally.
2. A method for migrating a storage system according to claim 1, characterized in that: It also includes migrating non-erasure coded data blocks in the original storage system into erasure coded data blocks.
3. A method for migrating a storage system as claimed in claim 2, characterized in that: The method also includes generating check data according to the erasure code data block.
4. A method for migrating a storage system according to claim 1, characterized in that: It also includes generating verification data according to the original storage system.
5. A system for migrating a storage system, characterized in that: include: Deployment module: used to deploy a new storage system on the original storage system; Information reading module: used to read the information including the file in the actual physical storage medium from the original storage system The original file description information of the starting address and ending address of the quality; Information import module: used to import the original file description information into the new storage system; Information reconstruction module: used to reconstruct new data description information based on the imported original file description information, including node identification number, storage device identification number, start address and end address, in the format of node:device:begin:end; Data service module: used to provide external data services.
6. The system according to claim 5, characterized in that Also includes: Data format conversion module: used to convert the non-erasure code data description information in the original storage system into the erasure code data description information.
7. The system according to claim 5, characterized in that Also includes: Verification code generation module: used to generate verification data according to the erasure code data block, or to generate verification data according to the original storage system.
8. An electronic device, characterized in that: The device comprises a processor and a memory, wherein: The memory is used to store executable programs; The processor is used to execute the executable program to implement any one of the methods for migrating a storage system according to claims 1-4.
