A method and device for rapid data migration

Abstract

The present invention proposes a method and device for rapid data migration. The method comprises: connecting to a target system via an SQLite database; obtaining the root page number and table field structure information of the target database; traversing the target database based on the root page number to collect all free pages and free blocks therein; determining the minimum valid data length in the target database based on the obtained field structure information, and rapidly screening the obtained free pages and free blocks based on the minimum valid data length; and migrating data on the free pages and free blocks that pass the screening. The present invention utilizes an SQLite database as the database for carrying the migrated data, significantly reducing the initial preparation work for data migration and the management work during the data migration process. Furthermore, by determining the minimum valid data length to screen distributed data in the target database, the method effectively increases the processing speed of invalid data, thereby significantly improving data migration efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer data migration, and in particular to a method and device for rapid data migration. Background Art

[0002] With the rapid advancement of communication technology and the expansion of specialized social divisions, interpersonal communication and social relationships are becoming increasingly systematized, networked, and refined. This has led to an increasing demand for server data capacity, and data recovery and migration are becoming increasingly prominent issues in this field. Rapid data migration has become a research hotspot and a challenge. Specifically, during data migration, databases contain a large number of free pages and blocks, which may contain valuable data. However, extracting this valuable data from these free pages and blocks is time-consuming.

[0003] Therefore, how to simply and quickly process free pages and free blocks to quickly complete data migration is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] In order to complete the data migration work of the target database more simply and quickly, one aspect of the present invention proposes a method for rapid data migration, which includes: docking the target system through an SQLite database (a lightweight database, an ACID-compliant relational database management system, which is contained in a relatively small C library); obtaining the root page number of the target database and the field structure information of the table; traversing the target database based on the root page number to collect all the free pages and free blocks therein; determining the minimum valid data length in the target database based on the obtained field structure information, and quickly screening the obtained free pages and free blocks based on the minimum valid data length; and migrating data on the free pages and free blocks that pass the screening.

[0005] In one or more implementations, the SQLite database is an embedded SQLite database, and the use of the SQLite database to connect to the target system includes: embedding a preset data migration program into the target system; in response to the target system executing the data migration program, adding the embedded SQLite database to the target system.

[0006] In one or more implementations, determining the minimum data length in the target database based on the acquired field structure information includes: determining the valid table field combination with the shortest byte length in the field structure information; and using the byte length occupied by the shortest valid table field combination as the minimum valid data length.

[0007] In one or more implementations, the quick screening of the obtained free pages and free blocks based on the minimum valid data length includes: in response to the byte length occupied by the free pages and free blocks being less than the minimum valid data length, skipping data extraction of the free pages and free blocks.

[0008] In one or more implementations, the data migration of the screened free pages and free blocks includes: determining the Type start bit of the table field in the free pages and free blocks; determining the theoretical length of the Type according to the variable-length integer storage rule of the Type; and saving the corresponding data in the free pages and free blocks into the embedded SQLite database based on the Type start bit of the table field and the theoretical length of the corresponding Type.

[0009] In a second aspect of the present invention, a data rapid migration device is proposed, comprising: a docking module, configured for docking with a target system via an SQLite database; an identification module, configured for obtaining the root page number of the target database and field structure information of a table; a collection module, configured for traversing the target database based on the root page number to collect all free pages and free blocks therein; a screening module, configured for determining the minimum valid data length in the target database based on the obtained field structure information, and quickly screening the obtained free pages and free blocks based on the minimum valid data length; and a migration module, configured for performing data migration on the free pages and free blocks that pass the screening.

[0010] In one or more implementations, the docking module is preset with an embedded SQLite database and a data migration program, and the docking module is further configured to: embed the preset data migration program into the target system; in response to the target system executing the data migration program, add the embedded SQLite database in the docking module to the target system.

[0011] In one or more implementations, the screening module is further configured to: determine the valid table field combination with the shortest byte length in the field structure information; and use the byte length occupied by the shortest valid table field combination as the minimum valid data length.

[0012] In one or more implementations, the screening module is further configured to: in response to the byte length occupied by the free page and the free block being less than the minimum valid data length, skip data extraction from the free page and the free block.

[0013] In one or more implementations, the migration module is further configured to: determine the Type start bit of the table field in the free page and the free block; determine the theoretical length of the Type according to the variable-length integer storage rule of the Type; and save the corresponding data in the free page and the free block into the embedded SQLite database based on the Type start bit of the table field and the theoretical length of the corresponding Type.

[0014] The beneficial effects of the present invention include: the beneficial effects of the present invention include: the present invention uses the SQLite database as the database for carrying the migration data, which greatly reduces the preliminary preparation work for data migration and the management work during the data migration process; and by determining the minimum valid data length to realize the screening of distributed data in the target database, effectively improving the processing speed of invalid data, thereby greatly improving the data migration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of a method for rapid data migration according to the present invention;

[0017] Figure 2 The figure is a structural diagram of a data rapid migration device of the present invention. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0019] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. Subsequent embodiments will not explain this one by one.

[0020] Figure 1 FIG. 1 is a flowchart of a method for rapid data migration according to the present invention. Figure 1As shown, the workflow of the data rapid migration method of the present invention includes: step S1, connecting to the target system through the SQLite database; step S2, obtaining the root page number of the target database and the field structure information of the table; step S3, traversing the target database based on the root page number to collect all the free pages and free blocks therein; step S4, determining the minimum valid data length in the target database according to the obtained field structure information, and quickly screening the obtained free pages and free blocks based on the minimum valid data length; step S5, migrating data on the free pages and free blocks that pass the screening.

[0021] In this embodiment, the present invention uses the SQLite database as the database for carrying the migration data, taking advantage of the fact that the SQLite database does not have a separate server process, does not require network configuration, and does not require management, which greatly facilitates the data migration work of the target database and greatly improves the data migration efficiency.

[0022] In one embodiment, the SQLite database of the present invention is an embedded SQLite database. Using the SQLite database to connect to the target system includes: embedding a preset data migration program into the target system; and adding the embedded SQLite database to the target system in response to the target system executing the data migration program.

[0023] In this embodiment, the SQLite database is designed as an embedded SQLite database, so that the program and database for controlling data migration operate in a unified address space. In this way, on the one hand, it helps to enhance the stability of the system and improve the efficiency of the system, and on the other hand, it also simplifies the docking work with the target server. Specifically, the data migration program can be embedded in the target system in advance. When the target system executes the data migration program, the embedded SQLite database is automatically added to the target system, thereby automatically completing the docking work with the target system, thereby effectively improving the docking efficiency.

[0024] In one embodiment, determining the minimum valid data length in the target database based on the acquired field structure information includes: determining the valid table field combination with the shortest byte length in the field structure information; and taking the byte length occupied by the shortest valid table field combination as the minimum valid data length.

[0025] Specifically, during the data migration process, the embedded SQLite database packages the data into data packets, the format of which is shown in Table 1 below:

[0026] Table 1 Data packet format

[0027]

[0028] The value of Payloadsize in the data packet header indicates the size of the data packet, Rowid represents the self-incrementing number of the database, and the unit contents Type1 to TypeN respectively indicate the type and length of Data1 to DataN.

[0029] Therefore, in a preferred embodiment, the three characteristic fields of payloadsize, rowId and headsizes are used as the shortest valid table field combination; since each table field needs to occupy at least one byte, the minimum valid data length is 4 bytes.

[0030] In a further embodiment, the obtained free pages and free blocks are quickly screened based on the minimum valid data length, including: in response to the byte length occupied by the free page or free block being less than the minimum valid data length, skipping data extraction from the free page or free block. Accordingly, in an optional implementation, when the size of the obtained free page or free block of target data is less than 4 bytes, data extraction from the free page or free block is skipped.

[0031] In another optional embodiment, all free pages and free blocks whose byte length is less than the minimum valid data length can be stored in a data table to be recovered, and data migration can be performed on them in a unified manner, so that data recovery can be performed based on the recovery data table after data migration is completed.

[0032] In a further embodiment, data migration is performed on the filtered free pages and free blocks, including: determining the Type start bit of the table field in the free pages and free blocks; determining the theoretical length of the Type according to the variable-length integer storage rule of the Type; and saving the corresponding data in the target database into the embedded SQLite database based on the Type start bit of the table field and the corresponding theoretical length of the Type.

[0033] The variable-length integer storage rule of Type is a default rule for setting table fields in the database. Those skilled in the art can roughly infer the length of the table fields based on the table fields. Common table fields and their lengths are shown in Table 2 below:

[0034] Table 2 Common table fields and their type lengths

[0035] Field Name type feature id Varchar server_path Varchar local_path Varchar size Varchar isdir Varchar 0x08 or 0x09 op_starttime DateTime 0x04 reversed1 Varchar 0x0D reversed2 Varchar 0x0D reversed3 Varchar 0x0D reversed4 Varchar 0x0D reversed5 Varchar 0x0D reversed6 Varchar 0x0D

[0036] In one embodiment, in the method of the present invention, a comparison table similar to Table 2 above may be constructed in advance, and when determining the shortest valid table field combination, a judgment may be made based on the comparison table.

[0037] In a second aspect of the present invention, a device for rapid data migration is provided. Figure 2 FIG. 1 is a structural diagram of a data rapid migration device of the present invention. Figure 2 As shown, the data rapid migration device of the present invention includes: a docking module 10, configured to dock with a target system through an SQLite database; an identification module 20, configured to obtain the root page number of the target database and the field structure information of the table; a collection module 30, configured to traverse the target database based on the root page number to collect all the free pages and free blocks therein; a screening module 40, configured to determine the minimum valid data length in the target database according to the obtained field structure information, and quickly screen the obtained free pages and free blocks based on the minimum valid data length; and a migration module 50, configured to perform data migration on the free pages and free blocks that pass the screening.

[0038] In one embodiment, the docking module 10 is preset with an embedded SQLite database and a data migration program, and the docking module 10 is further configured to: embed the preset data migration program into the target system; in response to the target system executing the data migration program, add the embedded SQLite database in the docking module to the target system.

[0039] In this embodiment, the SQLite database is designed as an embedded SQLite database, so that the program and database for controlling data migration operate in a unified address space. In this way, on the one hand, it helps to enhance the stability of the system and improve the efficiency of the system, and on the other hand, it also simplifies the docking work with the target server. Specifically, the data migration program can be embedded in the target system in advance. When the target system executes the data migration program, the embedded SQLite database is automatically added to the target system, thereby automatically completing the docking work with the target system, thereby effectively improving the docking efficiency.

[0040] In one embodiment, the screening module 40 is further configured to: determine the valid table field combination with the shortest byte length in the field structure information; and use the byte length occupied by the shortest valid table field combination as the minimum valid data length.

[0041] In one embodiment, the screening module 40 is further configured to: in response to the byte length occupied by the free pages and free blocks being less than the minimum valid data length, skip data extraction from the free pages and free blocks.

[0042] In one embodiment, the migration module 50 is further configured to: determine the Type start bit of the table field in the free pages and free blocks; determine the theoretical length of the Type according to the variable-length integer storage rule of the Type; and save the corresponding data in the target database into the embedded SQLite database based on the Type start bit of the table field and the corresponding theoretical length of the Type.

[0043] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless expressly limited to the singular.

[0044] It should be understood that, as used herein, the singular forms "a" and "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" is intended to include any and all possible combinations of one or more of the associated listed items.

[0045] The serial numbers of the embodiments disclosed in the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0046] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples. Within the spirit of the embodiments of the present invention, the technical features of the above embodiments or different embodiments may be combined, and there are many other variations of different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention.

Claims

1. A method for rapid data migration, characterized in that: The method comprises: Connect to the target system through the SQLite database; Get the root page number of the target database and the field structure information of the table; traverse the target database based on the root page number to collect all free pages and free blocks therein; Determining the minimum valid data length in the target database according to the acquired field structure information, and quickly screening the acquired free pages and free blocks based on the minimum valid data length; Perform data migration on the free pages and free blocks that pass the screening; The method of determining the minimum data length in the target database according to the acquired field structure information includes: determining a valid table field combination with the shortest byte length in the field structure information; and using the byte length occupied by the shortest valid table field combination as the minimum valid data length; The obtained free pages and free blocks are quickly screened based on the minimum valid data length, including: in response to the byte length occupied by the free pages and free blocks being less than the minimum valid data length, skipping data extraction of the free pages and free blocks.

2. The method for rapid data migration according to claim 1, wherein: The SQLite database is an embedded SQLite database, and the docking with the target system through the SQLite database includes: Embedding a preset data migration program into the target system; In response to the target system executing the data migration program, the embedded SQLite database is added to the target system.

3. The method for rapid data migration according to claim 1, wherein: The data migration of the free pages and free blocks that pass the screening includes: Determine the Type start bit of the table field in the free page and free block; Determine the theoretical length of Type according to the variable-length integer storage rules of Type; Based on the Type start bit of the table field and the theoretical length of the corresponding Type, the corresponding data in the free page and free block are saved in the embedded SQLite database.

4. A data rapid migration device, characterized in that: include: The docking module is configured to connect to the target system through the SQLite database; An identification module configured to obtain the root page number of a target database and field structure information of a table; A collection module configured to traverse the target database based on the root page number to collect all free pages and free blocks therein; a screening module configured to determine the minimum valid data length in the target database according to the acquired field structure information, and quickly screen the acquired free pages and free blocks based on the minimum valid data length; as well as A migration module is configured to migrate data on free pages and free blocks that pass the screening; The screening module is further configured to: determine the valid table field combination with the shortest byte length in the field structure information; and use the byte length occupied by the shortest valid table field combination as the minimum valid data length; The screening module is further configured to: in response to the byte length occupied by the free page and the free block being less than the minimum valid data length, skip data extraction from the free page and the free block.

5. The data rapid migration device according to claim 4, characterized in that: The docking module is pre-installed with an embedded SQLite database and a data migration program, and is further configured to: Embedding a preset data migration program into the target system; In response to the target system executing the data migration program, the embedded SQLite database in the docking module is added to the target system.

6. The data rapid migration device according to claim 5, characterized in that: The migration module is further configured to: Determine the Type start bit of the table field in the free page and free block; Determine the theoretical length of Type according to the variable-length integer storage rules of Type; Based on the Type start bit of the table field and the theoretical length of the corresponding Type, the corresponding data in the free page and free block are saved in the embedded SQLite database.

Images