Historical data migration method and system

Abstract

The present invention relates to the technical field of data processing, and disclosed are a historical data migration method and system. The method comprises: when data migration information sent by a historical database is acquired, establishing a communication connection between a target database and the historical database, and determining historical data to be migrated in the historical database; acquiring the data volume of the historical data to be migrated and the receivable data volume of the target database, and according to the data volume of said historical data and the receivable data volume of the target database, setting a migration instruction; on the basis of the migration instruction, migrating to the target database the historical data to be migrated; and once said historical data is migrated to the target database, determining an importance coefficient of the historical data in the historical database, and on the basis of the importance coefficient, classifying said historical data. The present invention solves the technical problem of inefficient historical data migration, thereby improving data migration efficiency, and effectively reducing data migration difficulty and data migration stress.

Description

Historical data migration method and system Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a historical data migration method and system. Background Art

[0002] In recent years, with the deepening of enterprise informatization, process-based management of core enterprise areas has become a crucial foundation for further improving informatization levels. Workflow systems, in a sense, as a form of informatization infrastructure, are occupying an increasingly important position in enterprise-level business management systems. To ensure that historical data in the original office system can be easily accessed and fully utilized at any time after the new system goes online, data migration is particularly important. The integrity and standardization of historical data play a key role in the practical application of system projects and the implementation of related work. At the same time, on the user side, how to best implement system switching in a single step to avoid interruptions in daily work due to system switching and the impact on daily enterprise operations and management is a key issue that must be considered.

[0003] In existing technologies, to migrate historical data, a production database and a historical database are typically created. The production database stores real-time business data. Historical data from the production database is then periodically migrated to the historical database to improve the performance of the production database. A timer is used to schedule batch operations, and when the timer arrives, data from the production database is migrated to the historical database. This process is repeated periodically. However, this existing historical data migration method has the following drawbacks: It is time-intensive, placing significant migration pressure on both the production and historical databases. Furthermore, with the continuous growth of business, this periodic migration method can no longer meet the migration efficiency requirements. Summary of the Invention

[0004] The embodiments of the present invention provide a historical data migration method and system to solve the technical problem in the prior art that the efficient migration of historical data cannot be guaranteed, the validity of the data cannot be guaranteed, and the pressure of data migration is increased.

[0005] To achieve the above object, the present invention provides a historical data migration method, which includes:

[0006] When data migration information sent by the historical database is obtained, a communication connection is established between the target database and the historical database, and the historical data to be migrated in the historical database is determined;

[0007] Acquiring the data volume of the historical data to be migrated and the receivable data volume of the target database, and setting a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database;

[0008] Migrating the historical data to be migrated to the target database based on the migration instruction;

[0009] After the historical data to be migrated is migrated to the target database, an importance coefficient of the historical data to be migrated in the historical database is determined, and the historical data to be migrated is classified based on the importance coefficient.

[0010] In one embodiment, before obtaining the amount of the historical data to be migrated and the amount of data that can be received by the target database, the method further includes:

[0011] Divide the historical data to be migrated into M batches of data, and import each batch of data into the migration temporary database in sequence.

[0012] Check whether the table structures of the data tables of the historical database, the data tables of the migration temporary database and the data tables of the target database are consistent,

[0013] If so, obtaining the amount of the historical data to be migrated;

[0014] If not, the table structures of the data tables of the historical database, the data tables of the migration temporary database, and the data tables of the target database are synchronized.

[0015] In one embodiment, when setting the migration instruction according to the amount of the historical data to be migrated and the amount of data that can be received by the target database, the method includes:

[0016] Determine whether the historical data to be migrated can be directly migrated to the target database based on the relationship between the data volume A of the historical data to be migrated and the receivable data volume a of the target database.

[0017] If the data volume A of the historical data to be migrated is less than or equal to the receivable data volume a of the target database, it is determined that the historical data to be migrated can be directly migrated to the target database;

[0018] If the data volume A of the historical data to be migrated is greater than the receivable data volume a of the target database, it is determined that the historical data to be migrated cannot be directly migrated to the target database.

[0019] In one embodiment, when it is determined that the historical data to be migrated can be directly migrated to the target database, the method further includes:

[0020] Calculate the data volume difference aA between the data volume A of the historical data to be migrated and the receivable data volume a of the target database,

[0021] When the data volume difference aA is greater than or equal to the data volume difference threshold, the historical data to be migrated is migrated to the target database;

[0022] When the data volume difference aA is less than the data volume difference threshold, the number of migrations of the historical data to be migrated is set according to the data volume difference aA, and the single migration amount of the historical data to be migrated is calculated according to the number of migrations and the data volume A, and the historical data to be migrated is migrated to the target database based on the number of migrations and the single migration amount.

[0023] In one embodiment, when setting the migration times of the historical data to be migrated according to the data amount difference aA, it includes:

[0024] A preset data amount difference matrix B is set to B(B1, B2, B3, B4), where B1 is a first preset data amount difference, B2 is a second preset data amount difference, B3 is a third preset data amount difference, and B4 is a fourth preset data amount difference, and B1 < B2 < B3 < B4;

[0025] Preset a migration number matrix C of the historical data to be migrated, set C(C1, C2, C3, C4, C5), where C1 is the first preset migration number, C2 is the second preset migration number, C3 is the third preset migration number, C4 is the fourth preset migration number, and C5 is the fifth preset migration number, and C1 < C2 < C3 < C4 < C5;

[0026] The number of migrations of the historical data to be migrated is set according to the relationship between the data amount difference aA and each preset data amount difference:

[0027] When A<B1, the first preset migration number C1 is selected as the migration number of the historical data to be migrated;

[0028] When B1≤A<B2, the second preset migration number C2 is selected as the migration number of the historical data to be migrated;

[0029] When B2≤A<B3, the third preset migration number C3 is selected as the migration number of the historical data to be migrated;

[0030] When B3≤A<B4, the fourth preset migration number C4 is selected as the migration number of the historical data to be migrated;

[0031] When B4≤A, the fifth preset migration number C5 is selected as the migration number of the historical data to be migrated.

[0032] In one embodiment, when the migration times of the historical data to be migrated is set to the i-th preset migration times Ci, i=1, 2, 3, 4, 5;

[0033] The single migration amount of the historical data to be migrated is calculated according to the following formula:

[0034] F=A / Ci;

[0035] Wherein, F is the single migration amount of the historical data to be migrated, A is the data volume of the historical data to be migrated, and Ci is the number of migrations of the historical data to be migrated.

[0036] In one embodiment, when it is determined that the historical data to be migrated cannot be directly migrated to the target database, the steps include:

[0037] Calculating a second data volume difference Aa between the data volume A of the historical data to be migrated and the receivable data volume a of the target database;

[0038] The target database is expanded according to the second data volume difference Aa, and the historical data to be migrated is migrated to the expanded target database.

[0039] In one embodiment, determining the importance coefficient of the historical data to be migrated in the historical database and classifying the historical data to be migrated based on the importance coefficient includes:

[0040] Acquire the number of times the historical data to be migrated is called in the historical database and the storage time of the historical data to be migrated in the historical database, wherein the number of times the historical data to be migrated is retrieved and used in the historical database;

[0041] Calculate the importance coefficient of the historical data to be migrated according to the number of calls and the storage time,

[0042] The importance coefficient of the historical data to be migrated is calculated according to the following formula:

[0043] P=W*ɑ+T*β;

[0044] Where P is the importance coefficient of the historical data to be migrated, W is the number of calls of the historical data to be migrated in the historical database, T is the storage time of the historical data to be migrated in the historical database, ɑ is the weight of the number of calls, β is the weight of the storage time, and ɑ + β = 1.

[0045] In one embodiment, the number of times J of the historical data to be migrated in the historical database is obtained, wherein the number of times the historical data to be migrated is referenced by the associated data;

[0046] Correct the important coefficient of the historical data to be migrated according to the number of times J the historical data to be migrated is cited in the historical database.

[0047] A preset citation count matrix G is set to G(G1, G2, G3, G4), where G1 is the first preset citation count, G2 is the second preset citation count, G3 is the third preset citation count, and G4 is the fourth preset citation count, and G1 < G2 < G3 < G4;

[0048] Preset an important coefficient correction coefficient matrix h for the historical data to be migrated, and set h(h1, h2, h3, h4, h5), where h1 is the first preset important coefficient correction coefficient, h2 is the second preset important coefficient correction coefficient, h3 is the third preset important coefficient correction coefficient, h4 is the fourth preset important coefficient correction coefficient, and h5 is the fifth preset important coefficient correction coefficient, and 0.8

[0049] The important coefficients of the historical data to be migrated are modified according to the relationship between the number of citations J and each preset number of citations:

[0050] When J<G1, the first preset important coefficient correction coefficient h1 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h1;

[0051] When G1≤J<G2, the second preset important coefficient correction coefficient h2 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h2;

[0052] When G2≤J<G3, the third preset important coefficient correction coefficient h3 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h3;

[0053] When G3≤J<G4, the fourth preset important coefficient correction coefficient h4 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h4;

[0054] When G4≤J, the fifth preset important coefficient correction coefficient h5 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h5.

[0055] In order to achieve the above object, the present invention further provides a historical data migration system, the system comprising:

[0056] ​a determination module, configured to establish a communication connection between a target database and the historical database upon acquiring data migration information sent by the historical database, and determine the historical data to be migrated in the historical database;

[0057] a setting module, configured to obtain the data volume of the historical data to be migrated and the receivable data volume of the target database, and set a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database;

[0058] A migration module, configured to migrate the historical data to be migrated to the target database based on the migration instruction;

[0059] The classification module is configured to determine, after the historical data to be migrated is migrated to the target database, an important coefficient of the historical data to be migrated in the historical database, and classify the historical data to be migrated based on the important coefficient.

[0060] The present invention provides a historical data migration method and system, which has the following beneficial effects compared with the existing technology:

[0061] The present invention discloses a historical data migration method and system. When data migration information sent by a historical database is obtained, a communication connection is established between a target database and the historical database, the historical data to be migrated in the historical database is determined, the data volume of the historical data to be migrated and the receivable data volume of the target database are obtained, a migration instruction is set according to the data volume of the historical data to be migrated and the receivable data volume of the target database, and the historical data to be migrated is migrated to the target database based on the migration instruction. After the historical data to be migrated is migrated to the target database, the importance coefficient of the historical data to be migrated in the historical database is determined, and the historical data to be migrated is classified based on the importance coefficient. The present invention solves the technical problem of being unable to efficiently migrate historical data, improves data migration efficiency, and effectively reduces the difficulty and pressure of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1 is a schematic diagram showing a process flow of a historical data migration method according to an embodiment of the present invention;

[0063] FIG2 shows a schematic structural diagram of a historical data migration system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0065] In the description of this application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0066] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0067] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0068] The following is a description of preferred embodiments of the present invention with reference to the accompanying drawings.

[0069] As shown in FIG1 , an embodiment of the present invention discloses a method for migrating historical data, the method comprising:

[0070] S110: When data migration information sent by the historical database is obtained, a communication connection is established between the target database and the historical database, and the historical data to be migrated in the historical database is determined.

[0071] In this embodiment, when historical data needs to be migrated, the historical database sends data migration information, establishes a communication connection between the target database and the historical database, and determines the historical data to be migrated in the historical database.

[0072] S120: Acquire the data volume of the historical data to be migrated and the receivable data volume of the target database, and set a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database.

[0073] In some embodiments of the present application, before obtaining the amount of the historical data to be migrated and the amount of receivable data of the target database, the method further includes:

[0074] Divide the historical data to be migrated into M batches of data, and import each batch of data into the migration temporary database in sequence.

[0075] Check whether the table structures of the data tables of the historical database, the data tables of the migration temporary database and the data tables of the target database are consistent,

[0076] If so, obtaining the amount of the historical data to be migrated;

[0077] If not, the table structures of the data tables of the historical database, the data tables of the migration temporary database, and the data tables of the target database are synchronized.

[0078] In this embodiment, the historical data to be migrated is randomly divided into M batches of data, and each batch of data is imported into the migration temporary database in turn, wherein the migration temporary database is located in the historical database, and it is detected whether the table structures of the data tables of the historical database, the data tables of the migration temporary database and the data tables of the target database are consistent, wherein the table structure refers to the consistency of the primary keys and column type lengths of the data tables of the historical database, the data tables of the migration temporary database and the data tables of the target database. If not, the table structures of the data tables of the historical database, the data tables of the migration temporary database and the data tables of the target database are synchronized, and then the next step is performed. The present invention can provide reliable support for migrating the historical data to be migrated, and ensure the smooth migration of the historical data to be migrated.

[0079] In some embodiments of the present application, when setting a migration instruction based on the amount of the historical data to be migrated and the amount of data that can be received by the target database, the process includes:

[0080] Determine whether the historical data to be migrated can be directly migrated to the target database based on the relationship between the data volume A of the historical data to be migrated and the receivable data volume a of the target database.

[0081] If the data volume A of the historical data to be migrated is less than or equal to the receivable data volume a of the target database, it is determined that the historical data to be migrated can be directly migrated to the target database;

[0082] If the data volume A of the historical data to be migrated is greater than the receivable data volume a of the target database, it is determined that the historical data to be migrated cannot be directly migrated to the target database.

[0083] In this embodiment, it is determined whether the target database can directly receive all the historical data to be migrated. When the data volume A of the historical data to be migrated is less than or equal to the target database's receivable data volume a, it is determined that the historical data to be migrated can be directly migrated to the target database. If the data volume A of the historical data to be migrated is greater than the target database's receivable data volume a, it is determined that the historical data to be migrated cannot be directly migrated to the target database. The present invention can avoid the situation where the target database has no storage space due to incomplete migration of the historical data to be migrated, and can ensure the normal migration of the historical data to be migrated, preventing the user's experience from being affected.

[0084] In some embodiments of the present application, when it is determined that the historical data to be migrated can be directly migrated to the target database, the method further includes:

[0085] Calculate the data volume difference aA between the data volume A of the historical data to be migrated and the receivable data volume a of the target database,

[0086] When the data volume difference aA is greater than or equal to the data volume difference threshold, the historical data to be migrated is migrated to the target database;

[0087] When the data volume difference aA is less than the data volume difference threshold, the number of migrations of the historical data to be migrated is set according to the data volume difference aA, and the single migration amount of the historical data to be migrated is calculated according to the number of migrations and the data volume A, and the historical data to be migrated is migrated to the target database based on the number of migrations and the single migration amount.

[0088] In this embodiment, the data volume difference aA between the data volume A of the historical data to be migrated and the receivable data volume a of the target database is calculated. When the data volume difference aA is greater than or equal to the data volume difference threshold, the historical data to be migrated is migrated to the target database. It should be understood that when the data volume difference aA is larger, it means that the receivable data volume of the target database is much larger than the data volume of the historical data to be migrated. Therefore, the historical data to be migrated can be directly migrated to the target database at one time. When the data volume difference aA is less than the data volume difference threshold, it means that the receivable data volume of the target database is close to the data volume of the historical data to be migrated. The number of migrations of the historical data to be migrated is set according to the data volume difference aA, and the single migration volume of the historical data to be migrated is calculated according to the number of migrations and the data volume A. The historical data to be migrated is migrated to the target database based on the number of migrations and the single migration volume. The present invention can further reduce the pressure on the target database to receive the historical data to be migrated, and ensure that the historical data to be migrated can be stably migrated to the target database.

[0089] In some embodiments of the present application, when setting the number of migrations of the historical data to be migrated according to the data amount difference aA, it includes:

[0090] A preset data amount difference matrix B is set to B(B1, B2, B3, B4), where B1 is a first preset data amount difference, B2 is a second preset data amount difference, B3 is a third preset data amount difference, and B4 is a fourth preset data amount difference, and B1 < B2 < B3 < B4;

[0091] Preset a migration number matrix C of the historical data to be migrated, set C(C1, C2, C3, C4, C5), where C1 is the first preset migration number, C2 is the second preset migration number, C3 is the third preset migration number, C4 is the fourth preset migration number, and C5 is the fifth preset migration number, and C1 < C2 < C3 < C4 < C5;

[0092] The number of migrations of the historical data to be migrated is set according to the relationship between the data amount difference aA and each preset data amount difference:

[0093] When A<B1, the first preset migration number C1 is selected as the migration number of the historical data to be migrated;

[0094] When B1≤A<B2, the second preset migration number C2 is selected as the migration number of the historical data to be migrated;

[0095] When B2≤A<B3, the third preset migration number C3 is selected as the migration number of the historical data to be migrated;

[0096] When B3≤A<B4, the fourth preset migration number C4 is selected as the migration number of the historical data to be migrated;

[0097] When B4≤A, the fifth preset migration number C5 is selected as the migration number of the historical data to be migrated.

[0098] In this embodiment, the number of migrations of the historical data to be migrated is set according to the relationship between the data volume difference aA and each preset data volume difference. This can avoid the phenomenon that, during the data migration process, the stability index of the target database changes significantly as the number of migrated data increases, or even causes the target database to stop responding.

[0099] In some embodiments of the present application, when the migration times of the historical data to be migrated is set to the i-th preset migration times Ci, i=1, 2, 3, 4, 5;

[0100] The single migration amount of the historical data to be migrated is calculated according to the following formula:

[0101] F=A / Ci;

[0102] Wherein, F is the single migration amount of the historical data to be migrated, A is the data volume of the historical data to be migrated, and Ci is the number of migrations of the historical data to be migrated.

[0103] In this embodiment, the migration stability of the historical data to be migrated can be further ensured by determining the number of migrations and the amount of migration per migration.

[0104] In some embodiments of the present application, when it is determined that the historical data to be migrated cannot be directly migrated to the target database, the steps include:

[0105] Calculating a second data volume difference Aa between the data volume A of the historical data to be migrated and the receivable data volume a of the target database;

[0106] The target database is expanded according to the second data volume difference Aa, and the historical data to be migrated is migrated to the expanded target database.

[0107] In this embodiment, a second data volume difference Aa between the data volume A of the historical data to be migrated and the receivable data volume a of the target database is calculated, and the target database is expanded according to the second data volume difference Aa. The specific expansion method is lengthy and mature, and will not be introduced in detail here. As long as the receivable data volume of the target database is greater than or equal to the data volume A of the historical data to be migrated, the present invention can ensure the smooth migration of the historical data to be migrated.

[0108] S130: Migrating the historical data to be migrated to the target database based on the migration instruction.

[0109] S140: After the historical data to be migrated is migrated to the target database, an importance coefficient of the historical data to be migrated in the historical database is determined, and the historical data to be migrated is classified based on the importance coefficient.

[0110] In some embodiments of the present application, determining the importance coefficient of the historical data to be migrated in the historical database and classifying the historical data to be migrated based on the importance coefficient includes:

[0111] Acquire the number of times the historical data to be migrated is called in the historical database and the storage time of the historical data to be migrated in the historical database, wherein the number of times the historical data to be migrated is retrieved and used in the historical database;

[0112] Calculate the importance coefficient of the historical data to be migrated according to the number of calls and the storage time,

[0113] The importance coefficient of the historical data to be migrated is calculated according to the following formula:

[0114] P=W*ɑ+T*β;

[0115] Where P is the importance coefficient of the historical data to be migrated, W is the number of calls of the historical data to be migrated in the historical database, T is the storage time of the historical data to be migrated in the historical database, ɑ is the weight of the number of calls, β is the weight of the storage time, and ɑ + β = 1.

[0116] In this embodiment, the number of calls of the historical data to be migrated in the historical database and the storage time of the historical data to be migrated in the historical database are obtained, wherein the storage time is the total storage time of the historical data to be migrated in the historical database, and the importance coefficient of the historical data to be migrated is calculated based on the number of calls and the storage time, and then the historical data to be migrated is classified according to the importance coefficient of the historical data to be migrated. When the importance coefficient is larger, the corresponding historical data to be migrated is more important, and vice versa, the less important it is. The present invention can classify the historical data to be migrated according to the importance coefficient, and provide convenience for subsequent data use.

[0117] In some embodiments of the present application, a reference count J of the historical data to be migrated in the historical database is obtained, wherein the reference count is the number of times the historical data to be migrated is referenced by associated data;

[0118] Correct the important coefficient of the historical data to be migrated according to the number of times J the historical data to be migrated is cited in the historical database.

[0119] A preset citation count matrix G is set to G(G1, G2, G3, G4), where G1 is the first preset citation count, G2 is the second preset citation count, G3 is the third preset citation count, and G4 is the fourth preset citation count, and G1 < G2 < G3 < G4;

[0120] Preset an important coefficient correction coefficient matrix h for the historical data to be migrated, and set h(h1, h2, h3, h4, h5), where h1 is the first preset important coefficient correction coefficient, h2 is the second preset important coefficient correction coefficient, h3 is the third preset important coefficient correction coefficient, h4 is the fourth preset important coefficient correction coefficient, and h5 is the fifth preset important coefficient correction coefficient, and 0.8

[0121] The important coefficients of the historical data to be migrated are modified according to the relationship between the number of citations J and each preset number of citations:

[0122] ​When J<G1, the first preset important coefficient correction coefficient h1 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h1;

[0123] When G1≤J<G2, the second preset important coefficient correction coefficient h2 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h2;

[0124] When G2≤J<G3, the third preset important coefficient correction coefficient h3 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h3;

[0125] When G3≤J<G4, the fourth preset important coefficient correction coefficient h4 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h4;

[0126] When G4≤J, the fifth preset important coefficient correction coefficient h5 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h5.

[0127] In this embodiment, the important coefficients of the historical data to be migrated are corrected according to the relationship between the number of citations J and each preset number of citations, wherein the number of citations is the number of times the historical data to be migrated is associated and used in other modules outside the historical database. The number of citations can reflect the degree of attention paid to the historical data to be migrated in the control or monitoring area, indicating that more than one function or module requires the target data, and the target data will also be used in other module databases. The more citations, the greater the correlation of the target data, the wider the scope, the greater the use, and the more important the target data. The present invention can further ensure the comprehensiveness and accuracy of the important coefficients of the historical data to be migrated, and provide accurate and reliable data support for the classification of the historical data to be migrated.

[0128] In order to further illustrate the technical idea of ​​the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.

[0129] Correspondingly, as shown in FIG2 , the present application further provides a historical data migration system, the system comprising:

[0130] a determination module, configured to establish a communication connection between a target database and the historical database upon acquiring data migration information sent by the historical database, and determine the historical data to be migrated in the historical database;

[0131] a setting module, configured to obtain the data volume of the historical data to be migrated and the receivable data volume of the target database, and set a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database;

[0132] A migration module, configured to migrate the historical data to be migrated to the target database based on the migration instruction;

[0133] The classification module is configured to determine, after the historical data to be migrated is migrated to the target database, an important coefficient of the historical data to be migrated in the historical database, and classify the historical data to be migrated based on the important coefficient.

[0134] In summary, the embodiment of the present invention establishes a communication connection between the target database and the historical database when obtaining data migration information sent by the historical database, determines the historical data to be migrated in the historical database, obtains the data volume of the historical data to be migrated and the receivable data volume of the target database, sets migration instructions according to the data volume of the historical data to be migrated and the receivable data volume of the target database, migrates the historical data to be migrated to the target database based on the migration instructions, and after the historical data to be migrated is migrated to the target database, determines the importance coefficient of the historical data to be migrated in the historical database, and classifies the historical data to be migrated based on the importance coefficient. The present invention solves the technical problem of being unable to efficiently migrate historical data, improves data migration efficiency, and effectively reduces the difficulty and pressure of data migration.

[0135] In the description of the above embodiments, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0136] Although the present invention has been described above with reference to exemplary embodiments, various modifications may be made and equivalent components may be substituted without departing from the scope of the present invention. In particular, the various features of the disclosed embodiments may be combined with one another in any manner, provided no structural conflicts exist. These combinations are not fully described in this specification solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but encompasses all technical solutions within the scope of the claims.

[0137] Those skilled in the art will understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will still be able to modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A historical data migration method, characterized in that: The method comprises: When data migration information sent by the historical database is obtained, a communication connection is established between the target database and the historical database, and the historical data to be migrated in the historical database is determined; Acquiring the data volume of the historical data to be migrated and the receivable data volume of the target database, and setting a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database; Migrating the historical data to be migrated to the target database based on the migration instruction; After the historical data to be migrated is migrated to the target database, an importance coefficient of the historical data to be migrated in the historical database is determined, and the historical data to be migrated is classified based on the importance coefficient.

2. The historical data migration method according to claim 1, characterized in that: Before obtaining the data volume of the historical data to be migrated and the receivable data volume of the target database, the method further includes: The historical data to be migrated is divided into M batches of data, and each batch of data is imported into the migration temporary database in sequence. Check whether the table structures of the data table of the historical database, the data table of the migration temporary database and the data table of the target database are consistent, If yes, then obtaining the data volume of the historical data to be migrated; If not, the table structures of the data table of the historical database, the data table of the migration temporary database and the data table of the target database are synchronized.

3. The historical data migration method according to claim 1, characterized in that: When the migration instruction is set according to the data volume of the historical data to be migrated and the receivable data volume of the target database, it includes: Determine whether the historical data to be migrated can be directly migrated to the target database based on the relationship between the data volume A of the historical data to be migrated and the receivable data volume a of the target database, If the data volume A of the historical data to be migrated is less than or equal to the receivable data volume a of the target database, it is determined that the historical data to be migrated can be directly migrated to the target database; If the data volume A of the historical data to be migrated is greater than the receivable data volume a of the target database, it is determined that the historical data to be migrated cannot be directly migrated to the target database.

4. The historical data migration method according to claim 3, characterized in that: When it is determined that the historical data to be migrated can be directly migrated to the target database, the method further includes: Calculate the data volume difference aA between the data volume A of the historical data to be migrated and the receivable data volume a of the target database, When the data volume difference aA is greater than or equal to the data volume difference threshold, the historical data to be migrated is migrated to the target database; When the data volume difference aA is less than the data volume difference threshold, the number of migrations of the historical data to be migrated is set according to the data volume difference aA, and the single migration amount of the historical data to be migrated is calculated according to the number of migrations and the data volume A, and the historical data to be migrated is migrated to the target database based on the number of migrations and the single migration amount.

5. The historical data migration method according to claim 4, characterized in that: When the migration times of the historical data to be migrated are set according to the data amount difference aA, it includes: A preset data volume difference matrix B is set to B (B1, B2, B3, B4), where B1 is a first preset data volume difference, B2 is a second preset data volume difference, B3 is a third preset data volume difference, B4 is a fourth preset data volume difference, and B1 < B2 < B3 < B4; Preset a migration number matrix C of the historical data to be migrated, set C (C1, C2, C3, C4, C5), where C1 is the first preset migration number, C2 is the second preset migration number, C3 is the third preset migration number, C4 is the fourth preset migration number, C5 is the fifth preset migration number, and C1<C2<C3<C4<C5; The number of migrations of the historical data to be migrated is set according to the relationship between the data volume difference aA and each preset data volume difference: When A<B1, the first preset migration number C1 is selected as the migration number of the historical data to be migrated; When B1≤A<B2, the second preset migration number C2 is selected as the migration number of the historical data to be migrated; When B2≤A<B3, the third preset migration number C3 is selected as the migration number of the historical data to be migrated; When B3≤A<B4, the fourth preset migration number C4 is selected as the migration number of the historical data to be migrated; When B4≤A, the fifth preset migration number C5 is selected as the migration number of the historical data to be migrated.

6. The historical data migration method according to claim 5, characterized in that: When the migration times of the historical data to be migrated is set to the i-th preset migration times Ci, i=1, 2, 3, 4, 5; The single migration amount of the historical data to be migrated is calculated according to the following formula: F = A / Ci; Wherein, F is the single migration amount of the historical data to be migrated, A is the data amount of the historical data to be migrated, and Ci is the number of migrations of the historical data to be migrated.

7. The historical data migration method according to claim 3, characterized in that: When it is determined that the historical data to be migrated cannot be directly migrated to the target database, the method includes: Calculate a second data volume difference Aa between the data volume A of the historical data to be migrated and the receivable data volume a of the target database; The target database is expanded according to the second data volume difference Aa, and the historical data to be migrated is migrated to the expanded target database.

8. The historical data migration method according to claim 1, characterized in that: When determining the importance coefficient of the historical data to be migrated in the historical database and classifying the historical data to be migrated based on the importance coefficient, the process includes: Acquire the number of times the historical data to be migrated is called in the historical database and the storage time of the historical data to be migrated in the historical database, wherein the number of times the historical data to be migrated is called and used in the historical database; Calculate the importance coefficient of the historical data to be migrated according to the number of calls and the storage time, The importance coefficient of the historical data to be migrated is calculated according to the following formula: P = W*ɑ+T*β; Among them, P is the importance coefficient of the historical data to be migrated, W is the number of calls of the historical data to be migrated in the historical database, T is the storage time of the historical data to be migrated in the historical database, ɑ is the weight of the number of calls, β is the storage time weight, and ɑ+β=1.

9. The historical data migration method according to claim 8, characterized in that: Also includes: Obtaining the number of times J of citations of the historical data to be migrated in the historical database, wherein the number of times of citations is the number of times the historical data to be migrated is cited by associated data; The important coefficient of the historical data to be migrated is modified according to the number of times J of citations of the historical data to be migrated in the historical database, A preset citation count matrix G is set to G (G1, G2, G3, G4), where G1 is the first preset citation count, G2 is the second preset citation count, G3 is the third preset citation count, G4 is the fourth preset citation count, and G1<G2<G3<G4; Preset an important coefficient correction coefficient matrix h of the historical data to be migrated, set h(h1, h2, h3, h4, h5), where h1 is the first preset important coefficient correction coefficient, h2 is the second preset important coefficient correction coefficient, h3 is the third preset important coefficient correction coefficient, h4 is the fourth preset important coefficient correction coefficient, h5 is the fifth preset important coefficient correction coefficient, and 0.8<h1<h2<h3<h4<h5<1.2; The important coefficients of the historical data to be migrated are modified according to the relationship between the reference number J and each preset reference number: When J<G1, the first preset important coefficient correction coefficient h1 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h1; When G1≤J<G2, the second preset important coefficient correction coefficient h2 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h2; When G2≤J<G3, the third preset important coefficient correction coefficient h3 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h3; When G3≤J<G4, the fourth preset important coefficient correction coefficient h4 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h4; When G4≤J, the fifth preset important coefficient correction coefficient h5 is selected to correct the important coefficient P of the historical data to be migrated, and the corrected important coefficient of the historical data to be migrated is P*h5.

10. A historical data migration system, characterized in that: The system comprises: A determination module, configured to establish a communication connection between a target database and the historical database when data migration information sent by the historical database is obtained, and determine the historical data to be migrated in the historical database; A setting module, used to obtain the data volume of the historical data to be migrated and the receivable data volume of the target database, and set a migration instruction according to the data volume of the historical data to be migrated and the receivable data volume of the target database; A migration module, configured to migrate the historical data to be migrated to the target database based on the migration instruction; The classification module is used to determine the importance coefficient of the historical data to be migrated in the historical database after the historical data to be migrated is migrated to the target database, and classify the historical data to be migrated based on the importance coefficient.

Images