Data migration method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of data processing, and provides a data migration method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring primary key data and non-primary key data of a to-be-migrated data set in a non-relational database; determining a primary column and a non-primary column of the non-primary key data; mapping the primary key data into a primary key of an initial data table of a relational database, mapping the primary column into a table field of the initial data table, and merging and mapping the non-primary column into an aggregated column of the initial data table to obtain mapping information of each field; and establishing a data table based on the mapping information to migrate the to-be-migrated data set to the relational database. According to the application, the non-primary column in the non-primary key data is merged and mapped, and the primary column is one-to-one mapped, so that the efficiency of migrating an NOSQL database to an RDBMS database is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data migration method and device, electronic equipment and storage medium. BACKGROUND

[0002] Non-relational database (Not Only SQL, NOSQL) and traditional relational database (Relational Database Management System, RDBMS) are the most popular database applications at present, each of which has different application scenarios and characteristics. At present, most of the database migration is from RDBMS to NOSQL, mainly to cope with new business application scenarios, so there is a demand for migration from NOSQL to RDBMS, but the existing solutions are relatively few, mainly including all column mapping and partial column mapping schemes.

[0003] The all column mapping method needs RDBMS to create many columns, which can easily cause the table structure to be large, and there are column number and total length limits in some RDBMS, so there are many restrictions in the conversion process. Many field values in NOSQL are empty, and the column data after mapping will also be empty, which is difficult to improve the storage utilization rate of the table and also increases the maintenance cost. If the migrated RDBMS uses a structured query language (Structured Query Language, MySQL) database, an inquiries and updates inside transactions (innodb) engine, the number of columns is too large, which will increase the probability of locking rows during data update operations, and when updating different columns in a row at the same time, there will be a delay in waiting for the lock. In addition, the minimum unit of the log in the database is a row of records, and each time the record log is updated, the record of each column is saved, which will generate a large amount of logs and waste storage space resources.

[0004] In the partial column mapping scheme, non-key data fields are discarded and only key fields are mapped, and the remaining fields are one-to-one mapped to the RDBMS table. This method often needs to modify the business logic associated with the data field, otherwise the absence of data will definitely affect the system business. The modification cycle of the business logic is generally long, and a large amount of time and human cost is needed for product demand modification evaluation, development workload estimation, test process follow-up.

[0005] In summary, the existing migration of NOSQL database to RDBMS database is low in efficiency. SUMMARY

[0006] Embodiments of the present application provide a data migration method and device, electronic equipment and storage medium to solve the technical problem of low efficiency in migrating a NOSQL database to an RDBMS database.

[0007] In a first aspect, embodiments of the present application provide a data migration method, comprising:

[0008] obtaining primary key data and non-primary key data of a to-be-migrated data set in a non-relational database;

[0009] determining a primary column and a non-primary column of the non-primary key data;

[0010] mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary column as a table field of the initial data table, and mapping the non-primary column as an aggregated column of the initial data table to obtain mapping information of each field;

[0011] establishing a data table based on the mapping information to migrate the to-be-migrated data set to the relational database.

[0012] In one embodiment, the determining of the primary column and the non-primary column of the non-primary key data comprises:

[0013] traversing all fields in the non-primary key data to obtain a proportion of each field;

[0014] if the proportion is less than a first set value, determining that the field is the non-primary column;

[0015] if the proportion is greater than or equal to the first set value, determining that the field is the primary column. In one embodiment, the determining of the primary column and the non-primary column of the non-primary key data further comprises:

[0016] traversing all fields in the non-primary key data to obtain a null value proportion of each field;

[0017] if the null value proportion is less than a second set value, determining that the field is the non-primary column;

[0018] if the null value proportion is greater than or equal to the second set value, determining that the field is the primary column.

[0019] In one embodiment, the mapping of the non-primary column as the aggregated column of the initial data table comprises:

[0020] determining an aggregation number of the non-primary column based on a data length of the aggregated column;

[0021] performing a combined mapping of the non-primary column based on the aggregation number to obtain the aggregated column of the initial data table.

[0022] In one embodiment, the establishing the data table based on the mapping information comprises:

[0023] generating an object relationship mapping configuration file based on the mapping information;

[0024] generating the data table based on the reverse conversion layer and the object relationship mapping configuration file; the reverse conversion layer is arranged between a service layer and a data access object layer.

[0025] In one embodiment, after the data set to be migrated is migrated to the relational database based on the establishing the data table based on the mapping information, the method further comprises:

[0026] obtaining a query request for data;

[0027] determining that the data to be queried belongs to a first target aggregated column based on the query request, and reversely converting the first target aggregated column into the non-primary column based on the reverse conversion layer and the object relationship mapping configuration file, so as to query the data to be queried based on the non-primary column.

[0028] In one embodiment, after the data set to be migrated is migrated to the relational database based on the establishing the data table based on the mapping information, the method further comprises:

[0029] obtaining an adjustment request for data;

[0030] if the adjustment request is to delete data in a second target aggregated column, determining the data to be deleted in the second target aggregated column based on the adjustment request, and deleting the data to be deleted;

[0031] if the adjustment request is to update the second target aggregated column, updating the second target aggregated column based on a storage type of data in the second target aggregated column.

[0032] In a second aspect, an embodiment of the present application provides a data migration device, comprising:

[0033] an obtaining module, configured to obtain primary key data and non-primary key data of a data set to be migrated in a non-relational database;

[0034] a determining module, configured to determine a primary column and a non-primary column in the non-primary key data;

[0035] a mapping module, configured to map the primary key data as a primary key of an initial data table of a relational database, map the primary column as a table field of the initial data table, and map the non-primary column as an aggregated column of the initial data table, so as to obtain mapping information of each field;

[0036] an importing module, configured to establish a data table based on the mapping information, so as to migrate the data set to be migrated to the relational database.

[0037] In a third aspect, an electronic device is provided, which includes a processor and a memory storing a computer program, and the processor implements the steps of the data migration method in the first aspect when executing the program.

[0038] In a fourth aspect, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the data migration method in the first aspect.

[0039] The data migration method, device, electronic device and storage medium provided by the embodiments of the present application can achieve the following effects. The embodiments of the present application obtain the primary key data and the non-primary key data of a data set to be migrated in a non-relational database; determine the primary column and the non-primary column of the non-primary key data; map the primary key data to the primary key of an initial data table of a relational database, map the primary column to the table field of the initial data table, and map the non-primary column to the aggregated column of the initial data table to obtain mapping information of each field; and establish a data table based on the mapping information to migrate the data set to be migrated to the relational database. The embodiments of the present application improve the efficiency of migrating the NOSQL database to the RDBMS database by mapping the non-primary column in the non-primary key data and one-to-one mapping the primary column. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is one of the flowcharts of the data migration method provided by the embodiments of the present application;

[0042] Figure 2 is a structural diagram of a traditional model-view-controller;

[0043] Figure 3 is a structural diagram of a model-view-controller with a reverse conversion layer provided by the embodiments of the present application;

[0044] Figure 4 is another flowchart of the data migration method provided by the embodiments of the present application;

[0045] Figure 5 is a structural diagram of the data migration device provided by the embodiments of the present application;

[0046] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0048] Figure 1 is one of flowcharts of a data migration method provided by an embodiment of the present application. With reference to Figure 1 , an embodiment of the present application provides a data migration method, comprising:

[0049] S100, acquiring primary key data and non-primary key data in a data set to be migrated in a non-relational database;

[0050] A non-relational database (NoSQL) does not use a traditional table, row and column structure to organize and store data, but uses various different data models to meet different types and needs of data storage and access. The non-relational database has flexible data models, elastic scalability, high performance and low latency, etc., and is suitable for processing large-scale, high-concurrency and semi-structured and complex relationship data scenarios.

[0051] The primary key data is a column or a group of columns in a database table, which is used to uniquely identify each row of data in the table.

[0052] The non-primary key data refers to the data of other columns or fields in the database table except the primary key. In the database table, the primary key data is used to uniquely identify each row of data, and the non-primary key data contains actual business data.

[0053] Connect the NOSQL database, read the data set to be migrated, and acquire specific data information; determine the primary key data in the data set to be migrated based on the attributes of the data set to be migrated, and the other data in the data set to be migrated is non-primary key data. For example, the document in the MongDB database always uses “_id” as the primary key, “_id” is generated by a timestamp, a machine code, a process identifier (PID) and a random number, which is used to uniquely identify a data column. This field can be mapped to a relational database as a primary key when mapping; because the field may be used as a query condition to query related information in the business.

[0054] S200, determining the primary column and the non-primary column of the non-primary key data;

[0055] The primary column and the non-primary column are set according to the importance or indispensability of the business logic. Alternatively, the primary column and the non-primary column of the non-primary key data are screened according to the custom rules.

[0056] In S300, the primary key data is mapped as a primary key of an initial data table of the relational database, the primary column is mapped as a table field of the initial data table, and the non-primary column is combined and mapped as an aggregate column of the initial data table, to obtain mapping information of each field.

[0057] The relational database is a database based on a relational model, which uses data tables to organize and store data. In the relational database, data is organized in the form of rows and columns, each data table contains multiple fields, each row represents a record, and each column represents an attribute.

[0058] The initial data table is a framework model of a data table in the relational database. After the primary key, the table field, the aggregate column and other data are migrated to the initial data table, the data table is obtained.

[0059] The primary key data is directly mapped as the primary key of the initial data table. The primary column and the non-primary column of the non-primary key data are analyzed according to the custom rules. The primary column is directly mapped into the table field of the initial data table. The non-primary column is combined and mapped into the aggregate column. The mapping information of each field is obtained based on the above data mapping.

[0060] In S400, a data table is established based on the mapping information, to migrate the to-be-migrated data set to the relational database.

[0061] The corresponding RDBMS database is connected. Based on the mapping information, the data in the to-be-migrated data set is imported into the initial data table in the form of the primary key, the table field and the aggregate column, the data table is established, and then the to-be-migrated data set is migrated to the relational database.

[0062] The data migration method provided in the embodiments of the application includes the following steps: obtaining primary key data and non-primary key data in a to-be-migrated data set in a non-relational database; determining a primary column and a non-primary column of the non-primary key data; mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary column as a table field of the initial data table, and combining and mapping the non-primary column as an aggregate column of the initial data table, to obtain mapping information of each field; and establishing a data table based on the mapping information, to migrate the to-be-migrated data set to the relational database. The non-primary column in the non-primary key data is combined and mapped, and the primary column is one-to-one mapped, so that the efficiency of migrating the NOSQL database to the RDBMS database is improved.

[0063] Based on the above embodiments, the primary column and the non-primary column of the non-primary key data are determined, including:

[0064] S210, traversing all fields in the non-primary key data to obtain the proportion of each field;

[0065] S220, if the proportion is less than the first set value, determining the field as a non-primary column;

[0066] S230, if the proportion is greater than or equal to the first set value, determining the field as a primary column.

[0067] The first set value is a proportion threshold set by human, which is set based on the importance or indispensability of business logic. In the embodiment of the present application, the first set value is 10%.

[0068] According to the custom rule, the non-primary columns of the non-primary key data are screened out, and the remaining data is the primary column. For example, by using the aggregation pipeline technology, all fields in the non-primary key data are traversed to obtain the proportion data of all fields. If the proportion of a field is less than 10%, these fields are identified as non-primary columns; if the proportion of a field is greater than or equal to 10%, these fields are identified as primary columns.

[0069] Based on the proportion data of each field, the primary column and the non-primary column in the non-primary key data are determined in the embodiment of the present application, which improves the efficiency of confirming the primary column and the non-primary column, and is further conducive to improving the efficiency of migrating the NOSQL database to the RDBMS database.

[0070] Based on the above embodiment, the primary column and the non-primary column of the non-primary key data are determined, and the method further comprises:

[0071] S240, traversing all fields in the non-primary key data to obtain the proportion of null values of each field;

[0072] S250, if the proportion of null values is less than the second set value, determining the field as a non-primary column;

[0073] S260, if the proportion of null values is greater than or equal to the second set value, determining the field as a primary column.

[0074] The second set value is a null value proportion threshold set by human, which is set based on the importance or indispensability of business logic. In the embodiment of the present application, the second set value is 10%.

[0075] By using the aggregation pipeline technology, all fields in the non-primary key data are traversed to obtain the proportion data of null values of all fields. If the proportion of null values of a field is less than 10%, these fields are identified as non-primary columns; if the proportion of null values of a field is greater than or equal to 10%, these fields are identified as primary columns.

[0076] The embodiment of the application determines the main column and the non-main column in the non-main key data based on the null proportion data of each field, improves the efficiency of confirming the main column and the non-main column, and further facilitates improving the efficiency of migrating the NOSQL database to the RDBMS database.

[0077] Based on the above embodiment, the non-main column is combined and mapped into the aggregated column of the initial data table, which includes:

[0078] In S310, the aggregated number of the non-main column is determined based on the data length of the aggregated column.

[0079] In S320, the non-main column is combined and mapped to obtain the aggregated column of the initial data table based on the aggregated number.

[0080] In order to make the performance of the RDBMS database and the table join query more convenient, the following needs to be considered when mapping the non-main column into the aggregated column: (1) how to set the number of the non-main column mapped into the aggregated column; (2) the total data length of the aggregated column is suitable for the interval; (3) how to process the index involved in the non-main column.

[0081] In order to ensure the conversion efficiency, the aggregated number of the non-main column is determined based on the data length of the aggregated column. The data length of the aggregated column is queried according to the aggregation pipeline technology of NOSQL. For example, when the NOSQL data is converted into the variable length character type (VARCHAR) in the structured query language (MySQL), it needs to be limited within 65535 strings. If the length exceeds the limit, it can be set to the text type (TEXT) or the medium text type (MEDIUMTEXT) or the long text type (LONGTEXT). In order to make the data length of the aggregated column within the limited range, the aggregated number of the non-main column needs to be limited.

[0082] For example, the data length of the aggregated column is not more than 65535 characters, and the data length of the non-main column is 13100. The aggregated number of the non-main column is 5. Five non-main columns are mapped into one aggregated column.

[0083] Further, for the index involved in the data of the non-main column, the non-main column is directly converted into the main column for one-to-one mapping. When the data of the non-main column involves the index, if it is directly stored in the aggregated column, the index cannot be used for data query in the RDBMS database. Therefore, the non-main column needs to be converted into the main column as a normal mapping.

[0084] The embodiment of the application determines the aggregated number of the non-main column, and then aggregates and maps the non-main column, thereby improving the efficiency of migrating the NOSQL database to the RDBMS database.

[0085] Based on the above embodiments, the data table is established based on the mapping information, comprising:

[0086] S410, generating an object relationship mapping configuration file based on the mapping information;

[0087] S420, generating a data table based on the reverse conversion layer and the object relationship mapping configuration file; the reverse conversion layer is arranged between the service layer and the data access object layer.

[0088] The reverse conversion layer is a conversion layer arranged in a Model-View-Controller (MVC). The architecture of a traditional MVC is as shown in Figure 2 Based on the traditional MVC, the operation process of outputting data externally is as follows: the request initiated by the controller layer passes through the business layer, initiates a query or change operation to the database, returns the query result, encapsulates the data through the business layer, and finally the result is output and displayed externally through the control layer. To ensure the minimum change of the outer layer of the system, a reverse conversion layer is arranged in the MVC structure, as shown in Figure 3 The main function of the reverse conversion layer is to perform reverse conversion of the multi-column mapping of NOSQL and the aggregated column of RDBMS in the operation of RDBMS database, so as to keep the service layer unchanged.

[0089] Object-Relational Mapping (ORM) is a technology for mapping between object models and relational databases. An ORM framework usually uses a configuration file to define the mapping relationship between entity classes and database tables, the settings of various properties, and other related configurations.

[0090] A data table is a structured data storage method in RDBMS, used to organize and store data. Each data table is composed of rows (records) and columns (fields), each row represents a specific data record, and each column represents a certain attribute or data field of the record. A data table includes: table name, column (field), row (record), primary key, foreign key, constraint, index.

[0091] Based on the mapping information, an ORM configuration file is generated, which records the names, types and mapping relationships of the conversion fields. The reverse conversion layer reads the ORM configuration file to obtain the mapping relationship between the attributes and the columns. The data to be migrated in NOSQL is converted into an ORM object, and a data table is generated based on the ORM object.

[0092] Further, the aggregation column is stored in the form of VARCHAR or TEXT or JavaScript Object Notation (JSON), and the attribute name and data type of the aggregation column remain unchanged from the type in the original NOSQL. For the complex embedded document in the NOSQL, the JSON format storage can be adopted.

[0093] To ensure the minimum change of the outer call of the system, the embodiment of the application adds a reverse conversion processing layer between the data access object layer (Data Access Object, DAO) and the service layer (Service), and refines the MVC architecture to the data calling flow at the actual code level, as shown in Figure 3 .

[0094] The embodiment of the application adds a reverse conversion layer between the DAO layer and the Service layer; on the basis of keeping the business logic of the service layer unchanged, reduces the error probability caused by the change; and improves the efficiency of the migration of the NOSQL database to the RDBMS database.

[0095] Based on the above embodiment, based on the mapping information, a data table is established to migrate the to-be-migrated data set to the relational database, and the method further includes:

[0096] S500, obtaining a query request of data;

[0097] S600, determining, based on the query request, that the to-be-queried data belongs to a first target aggregation column, and reversely converting the first target aggregation column into a non-primary column based on the reverse conversion layer and the object-relational mapping configuration file, to query the to-be-queried data based on the non-primary column.

[0098] After the data table is imported into the RDBMS database, the method further includes a query operation on the RDBMS database,

[0099] The first target aggregation column is an aggregation column containing to-be-queried data. The query operation includes that all data queried from the RDBMS database, the primary business column remains unchanged, and the data in the aggregation column (the first target aggregation column) is reversely converted according to the mapping of the ORM file. For example, the JSON string is converted into an ORM object, and the attributes in the JSON object are respectively split and mapped to the corresponding mapping columns in the Service layer of the original NOSQL, and then the to-be-queried data in the first target aggregation column is queried.

[0100] The embodiment of the application further performs a query operation on the relational database after importing the data table into the relational database, and improves the processing efficiency of the relational database.

[0101] The data table is established based on the mapping information to migrate the data set to be migrated to the relational database, and further includes:

[0102] S700, obtaining an adjustment request of data;

[0103] S800, if the adjustment request is to delete data in the second target aggregate column, determining data to be deleted in the second target aggregate column based on the adjustment request, and deleting the data to be deleted;

[0104] S900, if the adjustment request is to update the second target aggregate column, updating the second target aggregate column based on the storage type of the data in the second target aggregate column.

[0105] After the data table is imported into the RDBMS database, further including a deletion operation and an update operation on the RDBMS database.

[0106] The second target aggregate column includes an aggregate column to be deleted, and the deletion operation includes that the primary column data can be directly deleted, and since the aggregate column (the second target aggregate column) includes data that cannot be deleted, the second target aggregate column needs to be filtered to select and delete the data to be deleted.

[0107] The second target aggregate column includes an aggregate column to be updated, and the update operation includes that the data in the primary column is directly updated in the manner of the RDBMS, and the data in the aggregate column (the second target aggregate column) is updated according to the specific storage data type. For example, the JSON can be directly updated by using the structured query language (SQL), if it is other, such as the VARCHAR type, the update can be also realized by using the partial string operation SQL of the SQL, and the complex can be read to the program first and then updated.

[0108] The embodiments of the present application further perform a deletion operation and an update operation on the relational database after the data table is imported into the relational database, thereby improving the processing efficiency of the relational database.

[0109] In order to further analyze and describe the data migration method provided by the embodiments of the present application, the embodiments are specifically described as follows: Figure 4

[0110] (1) connecting a non-relational database by a program;

[0111] (2) reading data to be transferred in the non-relational database;

[0112] (3) determining the primary key data in the data to be transferred, and retaining the primary key data;

[0113] ​(4) For non-primary key data, according to the custom conversion rule, the primary column that needs one-to-one mapping and the non-primary column that needs combined mapping are analyzed, the primary column that needs one-to-one mapping is directly mapped into the table field in the RDBMS, and the non-primary column that needs combined mapping is mapped into the aggregate column;

[0114] (5) The database design mode of NOSQL is converted into the corresponding design mode of RDBMS, and the ORM configuration file of the scheme structure in RDBMS is completed according to the mapping field;

[0115] (6) The program connects the corresponding RDBMS database, and according to the database design mode in the ORM configuration file, the corresponding data table is established by reverse conversion;

[0116] (7) Complete the data batch conversion import.

[0117] The embodiments of the application avoid conversion failure caused by the column number and total length limit in part of the RDBMS by merging the multiple primary columns in NOSQL into the aggregate column in RDBMS. Reduce the number of table columns, improve the storage utilization and execution efficiency of the table, reduce the maintenance cost in the later period, and facilitate the business expansion in the later period. At the same time, a conversion layer is added on the Dao upper layer, the business logic of the service layer is kept unchanged, and the error probability caused by the change is reduced.

[0118] The data migration device provided by the embodiments of the application is described below. The data migration device described below can be correspondingly referred to the data migration method described above. Refer to Figure 5 , Figure 5 is a structural schematic diagram of the data migration device provided by the embodiments of the application. A data migration device comprises:

[0119] The acquisition module 501 is configured to acquire the primary key data and the non-primary key data in the non-relational database.

[0120] The determination module 502 is configured to determine the primary column and the non-primary column in the non-primary key data.

[0121] The mapping module 503 is configured to map the primary key data as the primary key of the initial data table of the relational database, map the primary column as the table field of the initial data table, and map the non-primary column as the aggregate column of the initial data table, to obtain the mapping information of each field.

[0122] The import module 504 is configured to establish the data table based on the mapping information, to migrate the to-be-migrated data set to the relational database.

[0123] The data migration device provided in the embodiments of the present application obtains the primary key data and the non-primary key data of a data set to be migrated in a non-relational database; determines the primary column and the non-primary column of the non-primary key data; maps the primary key data to the primary key of an initial data table of a relational database, maps the primary column to the table field of the initial data table, and maps the non-primary column to the aggregated column of the initial data table to obtain mapping information of each field; and establishes a data table based on the mapping information to migrate the data set to be migrated to the relational database. The embodiments of the present application improve the efficiency of migrating the NOSQL database to the RDBMS database by performing the combined mapping on the non-primary column in the non-primary key data and the one-to-one mapping on the primary column.

[0124] In one embodiment, the determining module 502 is configured to traverse all fields in the non-primary key data to obtain the proportion of each field; if the proportion is less than a first set value, the field is determined as a non-primary column; and if the proportion is greater than or equal to the first set value, the field is determined as a primary column.

[0125] In one embodiment, the determining module 502 is configured to traverse all fields in the non-primary key data to obtain the proportion of each field; if the proportion is less than a first set value, the field is determined as a non-primary column; and if the proportion is greater than or equal to the first set value, the field is determined as a primary column.

[0126] In one embodiment, the mapping module 503 is configured to determine the aggregation number of the non-primary column based on the data length of the aggregated column; and perform combined mapping on the non-primary column based on the aggregation number to obtain the aggregated column of the initial data table.

[0127] In one embodiment, the importing module 504 is configured to generate an object-relational mapping configuration file based on the mapping information; generate the data table based on the reverse conversion layer and the object-relational mapping configuration file; and set the reverse conversion layer between the service layer and the data access object layer.

[0128] In one embodiment, the importing module 504 is further configured to obtain a query request of data; determine, based on the query request, that the data to be queried belongs to a first target aggregated column; and reversely convert the first target aggregated column to the non-primary column based on the reverse conversion layer and the object-relational mapping configuration file, to query the data to be queried based on the non-primary column.

[0129] In one embodiment, the importing module 504 is further configured to obtain an adjustment request of data; if the adjustment request is to delete data in a second target aggregated column, determine the data to be deleted in the second target aggregated column based on the adjustment request, and delete the data to be deleted; and if the adjustment request is to update the second target aggregated column, update the second target aggregated column based on the storage type of the data in the second target aggregated column.

[0130] Figure 6 An example of an entity structure diagram of an electronic device is shown in FIG. 1.Figure 6 As shown, the electronic device can include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 complete mutual communication through the communication bus 640. The processor 610 can invoke the computer program in the memory 630 to perform the steps of the data migration method, for example, including:

[0131] Obtaining primary key data and non-primary key data of a to-be-migrated data set in a non-relational database; determining primary columns and non-primary columns of the non-primary key data; mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary columns as table fields of the initial data table, and mapping the non-primary columns as aggregated columns of the initial data table to obtain mapping information of each field; and establishing a data table based on the mapping information to migrate the to-be-migrated data set to the relational database.

[0132] In addition, the logical instructions in the memory 630 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0133] On the other hand, the embodiments of the present application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the steps of the data migration method provided by the above-mentioned embodiments, for example, including:

[0134] Obtaining primary key data and non-primary key data of a to-be-migrated data set in a non-relational database; determining primary columns and non-primary columns of the non-primary key data; mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary columns as table fields of the initial data table, and mapping the non-primary columns as aggregated columns of the initial data table to obtain mapping information of each field; and establishing a data table based on the mapping information to migrate the to-be-migrated data set to the relational database.

[0135] In another aspect, an embodiment of the present application also provides a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the data migration method provided by the above-mentioned embodiments, for example comprising:

[0136] obtaining primary key data and non-primary key data of a to-be-migrated data set in a non-relational database; determining primary columns and non-primary columns of the non-primary key data; mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary columns as table fields of the initial data table, and mapping the non-primary columns as aggregated columns of the initial data table to obtain mapping information of each field; and establishing a data table based on the mapping information to migrate the to-be-migrated data set to the relational database.

[0137] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to a magnetic storage (such as a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO), etc.), an optical storage (such as a CD, a DVD, a BD, a HVD, etc.), and a semiconductor storage (such as a ROM, an EPROM, an EEPROM, a non-volatile memory (NAND FLASH), a solid-state disk (SSD)), etc.

[0138] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0139] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such an understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data migration method, characterized by, The method comprises the following steps: obtaining primary key data and non-primary key data of a data set to be migrated in a non-relational database; determining a primary column and a non-primary column of the non-primary key data; mapping the primary key data as a primary key of an initial data table of a relational database, mapping the primary column as a table field of the initial data table, and mapping the non-primary column as an aggregated column of the initial data table to obtain mapping information of each field; establishing a data table based on the mapping information to migrate the data set to be migrated to the relational database.

2. The data migration method of claim 1, wherein, The determination of the primary column and the non-primary column of the non-primary key data comprises the following steps: traversing all fields in the non-primary key data to obtain a proportion of each field; if the proportion is less than a first set value, the field is determined as the non-primary column; if the proportion is greater than or equal to the first set value, the field is determined as the primary column.

3. The data migration method of claim 1, wherein, The determination of the primary column and the non-primary column of the non-primary key data further comprises the following steps: traversing all fields in the non-primary key data to obtain a proportion of empty values of each field; if the proportion of empty values is less than a second set value, the field is determined as the non-primary column; if the proportion of empty values is greater than or equal to the second set value, the field is determined as the primary column.

4. The data migration method of claim 1, wherein, The mapping of the non-primary column as the aggregated column of the initial data table comprises the following steps: determining an aggregation number of the non-primary column based on a data length of the aggregated column; based on the aggregation number, performing a combined mapping on the non-primary column to obtain the aggregated column of the initial data table.

5. The data migration method of claim 1, wherein, The establishment of the data table based on the mapping information comprises the following steps: generating an object-relational mapping configuration file based on the mapping information; generating the data table based on a reverse conversion layer and the object-relational mapping configuration file; the reverse conversion layer is arranged between a service layer and a data access object layer.

6. The data migration method of claim 5, wherein, After the data set to be migrated is migrated to the relational database based on the mapping information, the method further comprises the following steps: obtaining a query request of data; if it is determined that the data to be queried belongs to a first target aggregated column based on the query request, the first target aggregated column is reversely converted into the non-primary column based on the reverse conversion layer and the object-relational mapping configuration file, so that the data to be queried is queried based on the non-primary column.

7. The data migration method of claim 5, wherein, After the data set to be migrated is migrated to the relational database based on the mapping information, the method further comprises the following steps: obtaining an adjustment request of data; if the adjustment request is to delete data of a second target aggregated column, determining the data to be deleted in the second target aggregated column based on the adjustment request, and deleting the data to be deleted; if the adjustment request is to update the second target aggregated column, updating the second target aggregated column based on a storage type of data in the second target aggregated column.

8. A data migration apparatus, characterized by comprising: The method comprises the following steps: an obtaining module, configured to obtain primary key data and non-primary key data of a data set to be migrated in a non-relational database; a determining module, configured to determine a primary column and a non-primary column in the non-primary key data; The mapping module is configured to map the primary key data as a primary key of an initial data table of a relational database, map the primary column as a table field of the initial data table, and map the non-primary column combination as an aggregate column of the initial data table, to obtain mapping information of each field. The importing module is configured to establish a data table based on the mapping information, to migrate the to-be-migrated data set to the relational database.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the data migration method according to any one of claims 1 to 7 when executing the program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the data migration method according to any one of claims 1 to 7 when executed by the processor.

Images