Data migration method and device and readable storage medium

Abstract

The invention relates to a data migration method and device and a readable storage medium, and the method comprises the steps that a first data table in a non-relational database is acquired, and the first data table comprises a first table structure and first table content; analyzing and processing the first data table to obtain analysis data; performing mapping processing on first field information of each first field in the first data table according to a preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and second field information, the preset mapping rule comprises a preset mapping rule between the field information in the non-relational database and the field information in the relational database; determining a second table structure of a second data table according to the mapping relation, and converting the first table content into the second table structure according to the mapping relation to generate the second data table; and migrating the second data table to the relational database.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data migration method, device and readable storage medium. Background Art

[0002] At present, there are different data migration solutions for different data source switching scenarios. Due to the wide variety of data sources, heterogeneous data migration is more complicated and difficult. In the related art, the heterogeneous data migration solution from non-relational database to relational database is simple and not applicable to the situation where the data types in non-relational database are complex, and it is not universal. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a data migration method, device and readable storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a data migration method is provided, including:

[0005] Acquire a first data table in a non-relational database, where the first data table includes a first table structure and first table content;

[0006] Analyze and process the first data table to obtain analysis data;

[0007] According to a preset mapping rule and the analysis data, mapping processing is performed on the first field information of each first field in the first data table to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database;

[0008] Determine a second table structure of a second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table;

[0009] Migrate the second data table to the relational database.

[0010] In some embodiments, the analysis data includes the first field type quantity, the first field type, and the first field content of each of the first fields, the preset mapping rule includes a plurality of field type mapping rules, the first field information includes the first field type, the second field information includes the second field type, and the mapping relationship includes a field type mapping relationship;

[0011] The mapping process is performed on the first field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including:

[0012] For each of the first fields, a target field type mapping rule is obtained by matching the plurality of field type mapping rules according to the number of the first field types of the first field, the first field type, and at least one of the first field content, and the first field type is mapped according to the target field type mapping rule to obtain the field type mapping relationship between the first field type and the second field type.

[0013] In some embodiments, obtaining a target field type mapping rule by matching from the plurality of field type mapping rules according to at least one of the first field type quantity, the first field type, and the first field content of the first field includes:

[0014] When the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types;

[0015] When the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content.

[0016] In some embodiments, when the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types includes:

[0017] When the number of the first field types is greater than 1 and the number of the first field types is equal to 2, a first field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is equal to 2 is determined as the target field type mapping rule;

[0018] When the number of the first field types is greater than 1 and the number of the first field types is greater than 2, a second field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is greater than 2 is determined as the target field type mapping rule.

[0019] In some embodiments, when the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content, including:

[0020] When the number of the first field types is 1 and the first field type is a string type, determining a third field type mapping rule among the multiple field type mapping rules that matches the parsed type of the first field content as the target field type mapping rule;

[0021] When the number of the first field types is equal to 1 and the first field type is not the string type, a fourth field type mapping rule matching a non-string type among the multiple field type mapping rules is determined as the target field type mapping rule.

[0022] In some embodiments, the analysis data includes a first field name of each of the first fields, the preset mapping rule includes a field name mapping rule, the first field information includes the first field name, the second field information includes a second field name, and the mapping relationship includes a field name mapping relationship;

[0023] The mapping process is performed on the field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including:

[0024] For each of the first fields, the first field name of the first field is mapped according to the field name mapping rule to obtain the field name mapping relationship between the first field name and the second field name.

[0025] In some embodiments, the analysis data includes an index of each of the first fields, the preset mapping rule includes an index mapping rule, the first field information includes a first index, the second field information includes a second index, and the mapping relationship further includes an index mapping relationship;

[0026] The mapping process is performed on the first field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including:

[0027] For each of the first fields, a mapping process is performed on the first index of the first field according to the index mapping rule to obtain the index mapping relationship between the first index and the second index.

[0028] In some embodiments, the analysis data further includes a maximum content length of each of the first fields, and the method further includes:

[0029] When the second field type obtained after mapping the first field type of the first field is a string type, the target maximum content length corresponding to the first field is obtained from the analysis data, and the target maximum content length is expanded according to a preset expansion rule.

[0030] In some embodiments, after determining the second table structure of the second data table according to the mapping relationship, and converting the table content in the first data table into the second table structure according to the mapping relationship to generate the second data table, the method further includes:

[0031] An extended field is added to the second data table, where the extended field is used to record field exception information existing in the process of generating the second data table and the abnormal field that generates the field exception information.

[0032] In some embodiments, the analysis data includes the last write time of each first field. After analyzing and processing the first data table to obtain the analysis data, the method further includes:

[0033] When the total number of fields in the first data table is greater than a preset number, target fields in the first data table whose last written time is earlier than a preset time are deleted.

[0034] In some embodiments, determining a second table structure of a second data table according to the mapping relationship, and converting table content in the first data table into the second table structure according to the mapping relationship to generate the second data table includes:

[0035] Generate a mapping table according to the mapping relationship;

[0036] The second table structure is determined according to the mapping table, and the first table content is converted into the second table structure according to the mapping table to generate the second data table.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided a data migration device, including:

[0038] An acquisition module is configured to acquire a first data table in a non-relational database, wherein the first data table includes a first table structure and first table content;

[0039] An analysis module, configured to analyze and process the first data table to obtain analysis data;

[0040] a mapping module configured to perform mapping processing on the first field information of each first field in the first data table according to a preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database;

[0041] a generating module, configured to determine a second table structure of a second data table according to the mapping relationship, and to convert the first table content into the second table structure according to the mapping relationship to generate the second data table;

[0042] A migration module is configured to migrate the second data table to the relational database.

[0043] In some embodiments, the analysis data includes the first field type quantity, the first field type, and the first field content of each of the first fields, the preset mapping rule includes a plurality of field type mapping rules, the first field information includes the first field type, the second field information includes the second field type, and the mapping relationship includes a field type mapping relationship;

[0044] The mapping module is further configured to:

[0045] For each of the first fields, a target field type mapping rule is obtained by matching the plurality of field type mapping rules according to the number of the first field types of the first field, the first field type, and at least one of the first field content, and the first field type is mapped according to the target field type mapping rule to obtain the field type mapping relationship between the first field type and the second field type.

[0046] In some embodiments, the mapping module is further configured to:

[0047] When the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types;

[0048] When the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content.

[0049] In some embodiments, the mapping module is further configured to:

[0050] When the number of the first field types is greater than 1 and the number of the first field types is equal to 2, a first field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is equal to 2 is determined as the target field type mapping rule;

[0051] When the number of the first field types is greater than 1 and the number of the first field types is greater than 2, a second field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is greater than 2 is determined as the target field type mapping rule.

[0052] In some embodiments, the mapping module is further configured to:

[0053] When the number of the first field types is 1 and the first field type is a string type, determining a third field type mapping rule among the multiple field type mapping rules that matches the parsed type of the first field content as the target field type mapping rule;

[0054] When the number of the first field types is equal to 1 and the first field type is not the string type, a fourth field type mapping rule matching a non-string type among the multiple field type mapping rules is determined as the target field type mapping rule.

[0055] In some embodiments, the analysis data includes a first field name of each of the first fields, the preset mapping rule includes a field name mapping rule, the first field information includes the first field name, the second field information includes a second field name, and the mapping relationship includes a field name mapping relationship;

[0056] The mapping module is further configured to:

[0057] For each of the first fields, the first field name of the first field is mapped according to the field name mapping rule to obtain the field name mapping relationship between the first field name and the second field name.

[0058] In some embodiments, the analysis data includes an index of each of the first fields, the preset mapping rule includes an index mapping rule, the first field information includes a first index, the second field information includes a second index, and the mapping relationship further includes an index mapping relationship;

[0059] The mapping module is further configured to:

[0060] For each of the first fields, a mapping process is performed on the first index of the first field according to the index mapping rule to obtain the index mapping relationship between the first index and the second index.

[0061] In some embodiments, the analysis data further includes a maximum content length of each of the first fields, and the apparatus further includes:

[0062] The expansion module is configured to obtain the target maximum content length corresponding to the first field from the analysis data when the second field type obtained after mapping the first field type of the first field is a string type, and expand the target maximum content length according to a preset expansion rule.

[0063] In some embodiments, the apparatus further comprises:

[0064] The extension module is configured to add an extension field in the second data table, and the extension field is used to record the field exception information existing in the process of generating the second data table and the abnormal field that generates the field exception information.

[0065] In some embodiments, the analysis data includes the last write time of each first field, and the apparatus further includes:

[0066] The deleting module is configured to delete the target fields in the first data table whose last writing time is earlier than the preset time when the total number of fields in the first data table is greater than the preset number.

[0067] In some embodiments, the generating module is further configured to:

[0068] Generate a mapping table according to the mapping relationship;

[0069] The second table structure is determined according to the mapping table, and the first table content is converted into the second table structure according to the mapping table to generate the second data table.

[0070] According to a third aspect of an embodiment of the present disclosure, there is provided a data migration device, including:

[0071] processor;

[0072] a memory for storing processor-executable instructions;

[0073] Wherein, the processor is configured to:

[0074] Acquire a first data table in a non-relational database, where the first data table includes a first table structure and first table content;

[0075] Analyze and process the first data table to obtain analysis data;

[0076] According to a preset mapping rule and the analysis data, mapping processing is performed on the first field information of each first field in the first data table to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database;

[0077] Determine a second table structure of a second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table;

[0078] Migrate the second data table to the relational database.

[0079] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the data migration method provided in the first aspect of the present disclosure are implemented.

[0080] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by presetting mapping rules and analyzing data, the first field information of each first field in the first data table is mapped, and the mapping relationship between the first field information and the second field information is obtained, so that any field information in the first data table can be mapped to the second data table, and the versatility of data migration between non-relational databases and relational databases is improved. Moreover, by analyzing and processing the first data table, the detailed information of all data in the non-relational database can be parsed, the data details can be mastered, and the accuracy of mapping can be improved.

[0081] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0083] Figure 1 The figure is a flow chart of data migration in the related art according to an exemplary embodiment.

[0084] Figure 2 The figure is a flow chart of a data migration method according to an exemplary embodiment.

[0085] Figure 3It is a schematic diagram of a first field type mapping rule and a second field type mapping rule according to an exemplary embodiment.

[0086] Figure 4 It is a schematic diagram of a third field type mapping rule and a fourth field type mapping rule according to an exemplary embodiment.

[0087] Figure 5 is another flow chart of a data migration method according to an exemplary embodiment.

[0088] Figure 6 The figure is a block diagram of a data migration device according to an exemplary embodiment.

[0089] Figure 7 The invention is a block diagram showing a device for data migration according to an exemplary embodiment.

[0090] Figure 8 The invention is a block diagram showing a device for data migration according to an exemplary embodiment. DETAILED DESCRIPTION

[0091] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0092] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of data migration in a related art according to an exemplary embodiment. Figure 1 As shown, the related art implements data migration from a non-relational database to a relational database through the following scheme: first, the data in the non-relational database 110 is exported to a CSV (Comma Separated Value) file 120, and then the data in the CSV file 120 is imported into a relational database 130. However, the data migration scheme in the related art is simple, and is not applicable to the case where the data types in the non-relational database are complex, and is not universal.

[0093] For example, the data migration scheme in the related art is suitable for scenarios with simple and single data types, such as field types that only include basic types such as int (integer type) and string (string type). Once the data type is complex, such as when the data type includes deep-level json nesting and custom object types, there will be many problems, such as json (JavaScript Object Notation, JS object notation) type, hashmap (hash map) type, array (array) type conversion problems, garbled code problems, data null values ​​and other inconsistencies. Therefore, the data migration scheme in the related art is not applicable to situations where the data type is complex in non-relational databases and is not universal.

[0094] Figure 2 The present invention is a flowchart of a data migration method according to an exemplary embodiment, and the data migration method can be applied to a terminal or a server. In some embodiments, the data migration method of the present invention can be applied to any data migration scenario, for example, replacing an old system with a new one within an enterprise, and migrating the data of the old system to the new system, wherein the old system uses a non-relational database to store data, and the new system uses a relational database to store data. Figure 2 As shown, the data migration method may include the following steps.

[0095] Step 210: Acquire a first data table in a non-relational database, where the first data table includes a first table structure and first table content.

[0096] In some embodiments, the non-relational database may be a MongoDB database, and the first data table may be a collection in the MongoDB database. Data in the MongoDB database is stored in groups in a data set, which is referred to as a collection. The first data table may include at least one.

[0097] The first table structure is used to describe the framework of the first data table, and the first table structure at least includes field information of the first data table. The first table content may be a record in the first data table.

[0098] Step 220: Analyze and process the first data table to obtain analysis data.

[0099] Step 230, according to the preset mapping rules and analysis data, the first field information of each first field in the first data table is mapped to obtain a mapping relationship between the first field information and the second field information. The preset mapping rules include pre-set mapping rules between field information in a non-relational database and field information in a relational database.

[0100] In some embodiments, the relational database may be a MySQL database.

[0101] Step 240: determine a second table structure of the second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table.

[0102] Step 250: Migrate the second data table to the relational database.

[0103] In some embodiments, migrating the second data table to the relational database may include: storing the second data table in the relational database.

[0104] The disclosed embodiment maps the first field information of each first field in the first data table by presetting mapping rules and analyzing data, and obtains the mapping relationship between the first field information and the second field information, so that any field information in the first data table can be mapped to the second data table, thereby improving the versatility of data migration between non-relational databases and relational databases. Moreover, by analyzing and processing the first data table, detailed information of all data in the non-relational database can be parsed, data details can be mastered, and the accuracy of mapping can be improved.

[0105] In some embodiments, the analysis data may include the number of first field types, the first field type and the first field content of each first field, the preset mapping rules include multiple field type mapping rules, the first field information includes the first field type, the second field information includes the second field type, and the mapping relationship includes the field type mapping relationship; according to the preset mapping rules and the analysis data, the first field information of each first field is mapped to obtain a mapping relationship between the first field information and the second field information, including: for each first field, according to at least one of the number of first field types, the first field type and the first field content of the first field, a target field type mapping rule is matched from multiple field type mapping rules, and the first field type is mapped according to the target field type mapping rule to obtain a field type mapping relationship between the first field type and the second field type.

[0106] Since the first field in the non-relational database may correspond to multiple first field types, the target field type mapping rule is obtained by matching from multiple field type mapping rules according to at least one of the number of first field types, the first field type, and the first field content of the first field, and multiple field type mapping rules matching multiple information are preset, so that the first field of different data situations in the non-relational database can be accurately mapped to the second data table. For example, a first field of string type, or a first field storing multiple first field types, such as the first field storing int, double (double-precision floating point type) and string at the same time.

[0107] In some embodiments, a target field type mapping rule is matched from multiple field type mapping rules based on at least one of the number of first field types, the first field type, and the first field content of the first field, including: when the number of first field types is greater than 1, the target field type mapping rule is matched from multiple field type mapping rules based on the number of first field types; when the number of first field types is equal to 1, the target field type mapping rule is matched from multiple field type mapping rules based on the first field type and the first field content.

[0108] In some embodiments, when the number of first field types is greater than 1, a target field type mapping rule is obtained by matching from multiple field type mapping rules based on the first field type number, including: when the number of first field types is greater than 1, and the number of first field types is equal to 2, the first field type mapping rule in the multiple field type mapping rules that matches the situation where the number of first field types is equal to 2 is determined as the target field type mapping rule; when the number of first field types is greater than 1, and the number of first field types is greater than 2, the second field type mapping rule in the multiple field type mapping rules that matches the situation where the number of first field types is greater than 2 is determined as the target field type mapping rule.

[0109] For example, refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing a first field type mapping rule and a second field type mapping rule according to an exemplary embodiment. Figure 3 As shown, the first field type mapping rule includes: when the first field type includes null type and non-null type, the first field type of the first field is mapped to the non-null type of the second data table; when the first field type includes string type and objectid type, the first field type of the first field is mapped to the string type of the second data table; when the first field type is the other cases, the first field type of the first field is mapped to the json type of the second data table, and the other cases are the cases other than the aforementioned cases including null type and non-null type, and including string type and objectid type. And, as Figure 3 As shown, the second field type mapping rule includes: mapping any first field type of the first field to the json type in the second data table.

[0110] In some embodiments, when the number of first field types is equal to 1, a target field type mapping rule is matched from multiple field type mapping rules based on the first field type and the first field content, including: when the number of first field types is equal to 1 and the first field type is a string type, a third field type mapping rule that matches the parsed type of the first field content in the multiple field type mapping rules is determined as the target field type mapping rule; when the number of first field types is equal to 1 and the first field type is not a string type, a fourth field type mapping rule that matches the non-string type in the multiple field type mapping rules is determined as the target field type mapping rule.

[0111] For example, refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram showing a third field type mapping rule and a fourth field type mapping rule according to an exemplary embodiment. Figure 4 As shown, the third field type mapping rule includes: when the number of parsed types is equal to 1 and the parsed type is a string type, the first field type of the first field is mapped according to the maximum content length of the first field; when the number of parsed types is equal to 1 and the parsed type is a double type among non-string types, the first field type is mapped according to the preset precision requirement; when the number of parsed types is equal to 1 and the parsed type is a type other than double type among non-string types, the first field type is mapped to the parsed type; when the number of parsed types is greater than 1, any first field type of the first field is mapped to the json type in the second data table. For example, the parsed type can be int, boolean, etc.

[0112] In some embodiments, mapping the first field type of the first field according to the maximum content length of the first field may include: when the maximum content length exceeds 1500 characters, mapping the first field type to text; when the maximum content length exceeds 65535 characters, mapping the first field type to mediumtext.

[0113] In some embodiments, when the number of parsed types is equal to 1 and the parsed type is a double type among non-string types, the first field type is mapped according to a preset precision requirement, including: when the number of parsed types is equal to 1, the parsed type is a double type among non-string types, and the preset precision requirement is strong, the first field type is mapped to a string type; when the number of parsed types is equal to 1, the parsed type is a double type among non-string types, and the preset precision requirement is weak, the first field type is mapped to a decimal type.

[0114] like Figure 4 As shown, the fourth field type mapping rule includes: mapping the first field type int to int, mapping the first field type double to double, mapping the first field type long to biglnt, mapping the first field type json to json, mapping the first field type array to json, mapping the first field type boolean to tingint, mapping the first field type null to string, mapping the first field type date to dateTime, mapping the custom object type to json, and mapping the first field type objectid to string.

[0115] The disclosed embodiment sets a plurality of field type mapping rules, such as the first field type mapping rule to the fourth field type mapping rule, so as to accurately map the situation where there are a plurality of data types in the first field, and also realize the mapping of deeply nested JSON.

[0116] In some embodiments, the analysis data includes the first field name of each first field, the preset mapping rules include field name mapping rules, the first field information includes the first field name, the second field information includes the second field name, and the mapping relationship also includes a field name mapping relationship; according to the preset mapping rules and the analysis data, the first field information of each first field in the first data table is mapped to obtain a mapping relationship between the first field information and the second field information, including: for each first field, the first field name of the first field is mapped according to the field name mapping rules to obtain a field name mapping relationship between the first field name and the second field name.

[0117] In some embodiments, the field name mapping rules can be specifically set according to actual needs. For example, the field name mapping rules can include mapping the first field name to the corresponding English name according to business needs. Taking the non-relational database MongoDB as an example, the field names stored in MongoDB are all UUID (Universally Unique Identifier), which is a random number. There may be a problem that the field name is too long and the conversion of the second data table fails. Therefore, the field name is mapped to the corresponding English name. The English name is easy to understand, which can improve the readability and subsequent ease of use of the second data table.

[0118] In some embodiments, the method further includes: when the field names of the second data table obtained after mapping the field names of different first fields are the same, adding different suffixes to the same field names. Since the same field names may cause the problem of failure in converting the second data table, adding suffixes to the fields with the same names can distinguish the same field names and prevent failure in creating the second data table.

[0119] In some embodiments, the analysis data includes an index of each first field, the preset mapping rules include index mapping rules, the first field information includes a first index, the second field information includes a second index, and the mapping relationship also includes an index mapping relationship; according to the preset mapping rules and the analysis data, the first field information of each first field in the first data table is mapped to obtain a mapping relationship between the first field information and the second field information, including: for each first field, the first index of the first field is mapped according to the index mapping rules to obtain an index mapping relationship between the first index and the second index.

[0120] The present disclosure can implement mapping of three angles of the first data table, namely, field name, field type and index, through field type mapping rules, field name mapping rules and index mapping rules.

[0121] In some embodiments, the analysis data also includes the maximum content length of each first field, and the method also includes: when the second field type obtained after mapping the first field type of the first field is a string type, obtaining the target maximum content length corresponding to the first field from the analysis data, and expanding the target maximum content length according to a preset expansion rule.

[0122] In some embodiments, the target maximum content length is expanded according to a preset expansion rule, including: when the meaning of the first field is a Chinese description or a long log, the first field type of the first field is mapped to text, mediumtext or longtext; when the target maximum content length is less than 256, the target maximum content length is expanded to a target value, and the target value is determined by the product of 256 and the power of 2; when the target maximum content length exceeds 1500, the first field type of the first field is mapped to text; when the target maximum content length exceeds 65535, the first field type of the first field is mapped to mediumtext; when the target maximum content length exceeds 16777215, the first field type of the first field is mapped to longtext. In some embodiments, the target value can be 256*2 2 =1024.

[0123] In some embodiments, after determining the second table structure of the second data table according to the mapping relationship, and converting the table content in the first data table into the second table structure according to the mapping relationship to generate the second data table, the method also includes: adding an extended field in the second data table, and the extended field is used to record the field exception information existing in the process of converting the first data table to the second data table and the abnormal field that generates the field exception information.

[0124] In some embodiments, the field type of the extended field may be a mediumtext type. In some embodiments, field exception information may include: field type mismatch, field overlength, and field non-existence, etc. Field exception information may cause the second data table to fail to write and lose data. By adding an extended field, the failure to convert the second data table due to deviations in the analysis data of the first data table can be prevented. For example, the first field of the string type in the first data table currently scans out a maximum content length of 10. Although the capacity is expanded to 64, in fact, when the first field is written to the second data table, it is possible to write content with a length greater than 64, which will generate field exception information. By recording the field exception information and the abnormal field that generates the field exception information in the extended field, the second data table can be quickly repaired because the abnormal field is recorded, which does not affect the normal reading of the data in the second data table, while avoiding data loss.

[0125] In some embodiments, the analysis data also includes the last write time of each first field in the first data table. After analyzing and processing the first data table to obtain the analysis data, the method also includes: when the total number of fields in the first data table is greater than a preset number, deleting the target fields in the first data table whose last write time is earlier than the preset time.

[0126] In some embodiments, the preset number and the preset time can be specifically determined according to actual needs, for example, the preset number can be 200, and the preset time can be 2 years or 3 years, etc. For example, if the number of first fields in a single first data table is greater than 200, the conversion of the second data table may fail. Therefore, by deleting the target fields whose last write time is earlier than the preset time, the uncommonly used target fields can be deleted, thereby avoiding the conversion failure of the second data table.

[0127] In some embodiments, a second table structure of a second data table is determined according to a mapping relationship, and table contents in a first data table are converted into the second table structure according to the mapping relationship to generate a second data table, including: generating a mapping table according to the mapping relationship; determining the second table structure according to the mapping table, and converting the first table contents into the second table structure according to the mapping table to generate the second data table.

[0128] In some embodiments, the second table structure includes the second field information obtained after the mapping process. By determining the second table structure, the framework of the second data table can be obtained, and then the first table content is converted to the second table structure to obtain the second table content of the second data table, so that the second data table includes the corresponding second table structure and the second table content, forming a complete data table.

[0129] In the embodiment of the present disclosure, by generating a mapping table based on the mapping relationship, multiple mapping relationships obtained after mapping multiple first data tables can be stored in one table for easy management. The mapping relationship is stored in the mapping table. When the mapping relationship is needed to perform corresponding processing later (for example, the data synchronization described later), it is only necessary to obtain the corresponding mapping relationship from the mapping table without repeating the mapping process, thereby improving the efficiency of obtaining the mapping relationship.

[0130] As described above, the analysis data may include the number of first field types included in each first field in the first data table, the first field type of each first field, the first field content of each first field, the first field name of each first field, the first index of each first field, the maximum content length of each first field, and the last write time of each first field.

[0131] In this regard, the analysis data may also include more comprehensive information. For example, the analysis data may include: the scanning time of the first data table, the total number of documents included in the first data table, the collection name (or the first data table name), the total number of first fields in the first data table, the proportion of each first field in the first data table, the maximum length and minimum length of the content of each first field in the first data table, the number of the same first field type in the first data table, the distribution of first field types in the first data table, the earliest time and the latest time of each first field in the first data table, deep analysis of String type, the total number of indexes, index fields, and ascending and descending rules, etc.

[0132] String type deep analysis can be used to reflect whether the first field is completely mapped to int or boolean, etc., to perform more detailed type constraints on the data.

[0133] In some embodiments, converting the content of the first table into the second table structure according to the mapping table to generate the second data table may include: calling a data synchronization method, converting the content of the first table into the second table structure according to the mapping table, and generating the second data table. The data synchronization method may include full synchronization and incremental synchronization. For specific details about full synchronization and incremental synchronization, please refer to the relevant technology, which will not be repeated here.

[0134] In some embodiments, the method further includes: calling a data verification method, verifying the data in the first data table and the second data table, and outputting the verification result. The data verification method may include unifying the data in the first data table and the second data table into data of the same type; comparing the unified data of the same type, and outputting the verification result. For example, the first data table includes field contents of string type, and the second data table includes field contents of int type. The two can be unified into field contents of int type, and then the two int type field contents are compared. If they are consistent, it means that the data synchronization is correct. If they are inconsistent, it means that the data synchronization is incorrect. Through data verification, the data in the first data table and the second data table can be consistent, avoiding the situation where the data in the first data table is not obtained.

[0135] In order to illustrate the data migration method of the present disclosure in combination with data synchronization and data verification, the following will be combined with Figure 5 The data migration method disclosed in the present invention is described. Figure 5 As shown, the data migration method includes: obtaining a first data table 520 in a non-relational database 510; analyzing and processing the first data table 520 to obtain analysis data 530; mapping and processing the first field information of each first field in the first data table 520 according to a preset mapping rule and the analysis data 530 to obtain a mapping relationship between the first field information of the first data table 520 and the second field information of the second data table 550; generating a mapping table 540 according to the mapping relationship; determining the second table structure according to the mapping table 540, and converting the first table content into the second table structure according to the mapping table to generate the second data table 550, and migrating the second data table 550 to the relational database 560, wherein determining the second table structure according to the mapping table 540, and converting the first table content into the second table structure according to the mapping table to generate the second data table 550 can be a data synchronization process. After data synchronization, the data in the first data table 520 and the second data table 550 can be verified through a data verification process. About Figure 5 The relevant processes described in the previous article have been specifically explained in detail. For specific details, please refer to the relevant description in the previous article and will not be repeated here.

[0136] Figure 6 FIG. 1 is a block diagram of a data migration device according to an exemplary embodiment. Figure 6 , the data migration device 600 may include:

[0137] An acquisition module 610 is configured to acquire a first data table in a non-relational database, wherein the first data table includes a first table structure and first table content;

[0138] The analysis module 620 is configured to analyze and process the first data table to obtain analysis data;

[0139] A mapping module 630 is configured to perform mapping processing on the first field information of each first field in the first data table according to a preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database;

[0140] A generating module 640 is configured to determine a second table structure of a second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table;

[0141] The migration module 650 is configured to migrate the second data table to the relational database.

[0142] In some embodiments, the analysis data includes the first field type quantity, the first field type, and the first field content of each of the first fields, the preset mapping rule includes a plurality of field type mapping rules, the first field information includes the first field type, the second field information includes the second field type, and the mapping relationship includes a field type mapping relationship;

[0143] The mapping module 630 is further configured to:

[0144] For each of the first fields, a target field type mapping rule is obtained by matching the plurality of field type mapping rules according to the number of the first field types of the first field, the first field type, and at least one of the first field content, and the first field type is mapped according to the target field type mapping rule to obtain the field type mapping relationship between the first field type and the second field type.

[0145] In some embodiments, the mapping module 630 is further configured to:

[0146] When the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types;

[0147] When the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content.

[0148] In some embodiments, the mapping module 630 is further configured to:

[0149] When the number of the first field types is greater than 1 and the number of the first field types is equal to 2, a first field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is equal to 2 is determined as the target field type mapping rule;

[0150] When the number of the first field types is greater than 1 and the number of the first field types is greater than 2, a second field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is greater than 2 is determined as the target field type mapping rule.

[0151] In some embodiments, the mapping module 630 is further configured to:

[0152] When the number of the first field types is 1 and the first field type is a string type, determining a third field type mapping rule among the multiple field type mapping rules that matches the parsed type of the first field content as the target field type mapping rule;

[0153] When the number of the first field types is equal to 1 and the first field type is not the string type, a fourth field type mapping rule matching a non-string type among the multiple field type mapping rules is determined as the target field type mapping rule.

[0154] In some embodiments, the analysis data includes a first field name of each of the first fields, the preset mapping rule includes a field name mapping rule, the first field information includes the first field name, the second field information includes a second field name, and the mapping relationship includes a field name mapping relationship;

[0155] The mapping module 630 is further configured to:

[0156] For each of the first fields, the first field name of the first field is mapped according to the field name mapping rule to obtain the field name mapping relationship between the first field name and the second field name.

[0157] In some embodiments, the analysis data includes an index of each of the first fields, the preset mapping rule includes an index mapping rule, the first field information includes a first index, the second field information includes a second index, and the mapping relationship further includes an index mapping relationship;

[0158] The mapping module 630 is further configured to:

[0159] For each of the first fields, a mapping process is performed on the first index of the first field according to the index mapping rule to obtain the index mapping relationship between the first index and the second index.

[0160] In some embodiments, the analysis data further includes a maximum content length of each of the first fields, and the apparatus further includes:

[0161] The expansion module is configured to obtain the target maximum content length corresponding to the first field from the analysis data when the second field type obtained after mapping the first field type of the first field is a string type, and expand the target maximum content length according to a preset expansion rule.

[0162] In some embodiments, the apparatus further comprises:

[0163] The extension module is configured to add an extension field in the second data table, and the extension field is used to record the field exception information existing in the process of generating the second data table and the abnormal field that generates the field exception information.

[0164] In some embodiments, the analysis data includes the last write time of each first field, and the apparatus further includes:

[0165] The deleting module is configured to delete the target fields in the first data table whose last writing time is earlier than the preset time when the total number of fields in the first data table is greater than the preset number.

[0166] In some embodiments, the generating module 640 is further configured to:

[0167] Generate a mapping table according to the mapping relationship;

[0168] The second table structure is determined according to the mapping table, and the first table content is converted into the second table structure according to the mapping table to generate the second data table.

[0169] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0170] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and the program instructions, when executed by a processor, implement the steps of the data migration method provided by the present disclosure.

[0171] Figure 77 is a block diagram of an apparatus 700 for data migration according to an exemplary embodiment. For example, the apparatus 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0172] Reference Figure 7 , the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output interface 712 , a sensor component 714 , and a communication component 716 .

[0173] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0174] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0175] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0176] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0177] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0178] The input / output interface 712 provides an interface between the processing component 702 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0179] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0180] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0181] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0183] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above data migration method when executed by the programmable device.

[0184] Figure 8 8 is a block diagram of a device 800 for data migration according to an exemplary embodiment. For example, the device 800 may be provided as a server. Figure 8 The apparatus 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions, such as an application, that can be executed by the processing component 822. The application stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned data migration method.

[0185] The device 800 may also include a power supply component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output interface 858. The device 800 may operate based on an operating system stored in the memory 832, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0187] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data migration method, characterized in that: include: Acquire a first data table in a non-relational database, where the first data table includes a first table structure and first table content; Analyze and process the first data table to obtain analysis data; According to a preset mapping rule and the analysis data, mapping processing is performed on the first field information of each first field in the first data table to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database; Determine a second table structure of a second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table; Migrate the second data table to the relational database.

2. The data migration method according to claim 1, characterized in that: The analysis data includes the first field type quantity, the first field type and the first field content of each of the first fields, the preset mapping rule includes a plurality of field type mapping rules, the first field information includes the first field type, the second field information includes the second field type, and the mapping relationship includes a field type mapping relationship; The mapping process is performed on the first field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including: For each of the first fields, a target field type mapping rule is obtained by matching the plurality of field type mapping rules according to the number of the first field types of the first field, the first field type, and at least one of the first field content, and the first field type is mapped according to the target field type mapping rule to obtain the field type mapping relationship between the first field type and the second field type.

3. The data migration method according to claim 2, characterized in that: The step of matching the target field type mapping rule from the plurality of field type mapping rules according to at least one of the first field type quantity, the first field type, and the first field content of the first field includes: When the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types; When the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content.

4. The data migration method according to claim 3, characterized in that: When the number of the first field types is greater than 1, obtaining the target field type mapping rule by matching the plurality of field type mapping rules according to the number of the first field types includes: When the number of the first field types is greater than 1 and the number of the first field types is equal to 2, a first field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is equal to 2 is determined as the target field type mapping rule; When the number of the first field types is greater than 1 and the number of the first field types is greater than 2, a second field type mapping rule among the multiple field type mapping rules that matches the situation where the number of the first field types is greater than 2 is determined as the target field type mapping rule.

5. The data migration method according to claim 3, characterized in that: When the number of the first field types is equal to 1, the target field type mapping rule is obtained by matching from the multiple field type mapping rules according to the first field type and the first field content, including: When the number of the first field types is 1 and the first field type is a string type, determining a third field type mapping rule among the multiple field type mapping rules that matches the parsed type of the first field content as the target field type mapping rule; When the number of the first field types is equal to 1 and the first field type is not the string type, a fourth field type mapping rule matching a non-string type among the multiple field type mapping rules is determined as the target field type mapping rule.

6. The data migration method according to any one of claims 1 to 5, characterized in that: The analysis data includes a first field name of each of the first fields, the preset mapping rule includes a field name mapping rule, the first field information includes the first field name, the second field information includes a second field name, and the mapping relationship includes a field name mapping relationship; The mapping process is performed on the field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including: For each of the first fields, the first field name of the first field is mapped according to the field name mapping rule to obtain the field name mapping relationship between the first field name and the second field name.

7. The data migration method according to any one of claims 1 to 5, characterized in that: The analysis data includes an index of each of the first fields, the preset mapping rule includes an index mapping rule, the first field information includes a first index, the second field information includes a second index, and the mapping relationship further includes an index mapping relationship; The mapping process is performed on the first field information of each first field in the first data table according to the preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, including: For each of the first fields, a mapping process is performed on the first index of the first field according to the index mapping rule to obtain the index mapping relationship between the first index and the second index.

8. The data migration method according to claim 2-5, characterized in that: The analysis data further includes a maximum content length of each of the first fields, and the method further includes: When the second field type obtained after mapping the first field type of the first field is a string type, the target maximum content length corresponding to the first field is obtained from the analysis data, and the target maximum content length is expanded according to a preset expansion rule.

9. The data migration method according to claim 1, characterized in that: After determining the second table structure of the second data table according to the mapping relationship, and converting the table content in the first data table into the second table structure according to the mapping relationship to generate the second data table, the method further includes: An extended field is added to the second data table, where the extended field is used to record field exception information existing in the process of generating the second data table and the abnormal field that generates the field exception information.

10. The data migration method according to claim 1, characterized in that: The analysis data includes the last write time of each of the first fields. After the first data table is analyzed and processed to obtain the analysis data, the method further includes: When the total number of fields in the first data table is greater than a preset number, target fields in the first data table whose last written time is earlier than a preset time are deleted.

11. The data migration method according to claim 1, characterized in that: The determining a second table structure of the second data table according to the mapping relationship, and converting the table content in the first data table into the second table structure according to the mapping relationship to generate the second data table includes: Generate a mapping table according to the mapping relationship; The second table structure is determined according to the mapping table, and the first table content is converted into the second table structure according to the mapping table to generate the second data table.

12. A data migration device, characterized in that: include: An acquisition module is configured to acquire a first data table in a non-relational database, wherein the first data table includes a first table structure and first table content; An analysis module, configured to analyze and process the first data table to obtain analysis data; a mapping module configured to perform mapping processing on the first field information of each first field in the first data table according to a preset mapping rule and the analysis data to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database; a generating module, configured to determine a second table structure of a second data table according to the mapping relationship, and to convert the first table content into the second table structure according to the mapping relationship to generate the second data table; A migration module is configured to migrate the second data table to the relational database.

13. A data migration device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire a first data table in a non-relational database, where the first data table includes a first table structure and first table content; Analyze and process the first data table to obtain analysis data; According to a preset mapping rule and the analysis data, mapping processing is performed on the first field information of each first field in the first data table to obtain a mapping relationship between the first field information and the second field information, wherein the preset mapping rule includes a preset mapping rule between the field information in the non-relational database and the field information in the relational database; Determine a second table structure of a second data table according to the mapping relationship, and convert the first table content into the second table structure according to the mapping relationship to generate the second data table; Migrate the second data table to the relational database.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the data migration method described in any one of claims 1 to 11 are implemented.

Images