Data migration method and related device based on high-dimensional strategy optimization and full-link evidence storage

Abstract

The invention provides a data migration method based on high-dimensional strategy optimization and full-link evidence storage and a related device, which are suitable for a data migration scene between heterogeneous databases. According to the method, firstly, a multi-dimensional time decay tensor is constructed to express system state features, and then an optimal migration strategy meeting resource constraints is generated based on a reinforcement learning framework. And in a migration execution stage, selecting a full-amount, incremental or mixed migration mode according to a strategy, and performing semantic mapping conversion on the SQL statement to adapt to a target database structure. And after migration is completed, performing weighted verification on the source target data through an integral consistency function, and finally writing an optimal migration strategy and a verification result into the block chain to form a non-tampering migration log. The method has high adaptability, high consistency guarantee and whole-process traceability, and can effectively improve data migration efficiency and credibility in a complex database environment.

Description

Technical Field

[0001] This invention relates to the field of database migration technology, specifically a data migration method and related apparatus based on high-dimensional strategy optimization and end-to-end evidence storage. Background Technology

[0002] Against the backdrop of the rapid development of big data and cloud computing, heterogeneous deployment of database systems, multi-site disaster recovery, and business restructuring are becoming increasingly frequent, leading to a continuous increase in enterprises' and institutions' reliance on data migration technologies. Database migration typically involves multiple stages, including data structure replication, content synchronization, semantic compatibility, and historical data verification, and is one of the key supporting technologies for ensuring business continuity, system scalability, and data governance compliance.

[0003] However, traditional data migration solutions generally rely on manually setting migration batches and rules, lacking the ability to dynamically perceive and optimize real-time business status, resource bottlenecks, and data sensitivity. When faced with large-scale heterogeneous databases, real-time change scenarios, or resource-constrained environments, problems such as migration timeouts, high failure rates, or inconsistencies in target data are prone to occur. Furthermore, most systems lack end-to-end chain-based auditable mechanisms, resulting in a lack of reliable records of the migration process and risks such as difficulties in operational traceability and unclear liability definition.

[0004] In recent years, some studies have attempted to improve migration efficiency by introducing machine learning, policy scheduling, or log auditing, but several shortcomings remain. On the one hand, most existing methods use single-dimensional feature modeling, lacking high-dimensional representation and dynamic weight modeling of the "multi-dimensional, multi-temporal, and multi-source" states in data migration systems. On the other hand, there is still a lack of a unified methodology that forms a logical closed loop between policy formulation, consistency assessment, SQL semantic transformation, and migration process recording, resulting in severe fragmentation of migration solutions and difficulty in end-to-end deployment.

[0005] Therefore, there is an urgent need for a data migration method with high-dimensional state representation capabilities, policy adaptive learning capabilities, consistency integral verification capabilities, and end-to-end trusted recording capabilities. This method should achieve breakthroughs over existing migration technologies in terms of method structure, computational expression, and system implementation, so as to better support the real needs of government and enterprise units, the financial industry, and operators for complex data migration scenarios. Summary of the Invention

[0006] This invention provides a data migration method and related apparatus based on high-dimensional strategy optimization and end-to-end evidence storage, solving the problems of existing database migration processes, such as strategy dependence on manual setting, weak environmental awareness, difficulty in quantifying and verifying the consistency of migrated data, and lack of full-process auditability. This method constructs a multi-dimensional state tensor, uses reinforcement learning to generate migration strategies, combines semantic graph mapping to achieve automatic SQL statement conversion, employs an integral consistency function to perform weighted verification of source and target data, and writes key elements of the migration process into the blockchain to form an immutable evidence storage log, thereby achieving an efficient, stable, and traceable closed-loop data migration process.

[0007] A data migration method based on high-dimensional strategy optimization and end-to-end evidence storage includes the following steps:

[0008] Step (1) Multidimensional tensor feature extraction: Collect dynamic features of the migration environment, such as resource utilization rate, data change frequency, and migration window, and generate a state input tensor by time decay weighting.

[0009] Step (2) Reinforcement policy generation: Based on the state input tensor output in step (1), the optimal transfer policy is generated through reinforcement learning;

[0010] Step (3) Construction of migration dataset: Based on the optimal migration strategy generated in step (2), select a data migration mode to generate a migration dataset. The data migration mode includes full or incremental migration mode or a hybrid migration mode of full or incremental.

[0011] Step (4) SQL semantic mapping conversion: Based on the migration dataset output in step (3), the source database SQL statements are converted into target SQL in a format compatible with the target database through the semantic mapping function, so as to realize the equivalent conversion of SQL syntax between different databases and maintain the logical consistency before and after migration;

[0012] Step (5) Consistency score verification: Perform a weighted integral verification on the target SQL transformed in step (4) and calculate the consistency score I;

[0013] Step (6) Blockchain log storage: Write the optimal migration strategy from step (2) and the consistency score I from step (5) into the blockchain to build an auditable and tamper-proof migration chain and complete the full-link storage.

[0014] Furthermore, in step (1), the state input tensor is generated by time decay weighting, as shown in the following formula:

[0015] ;

[0016] in, For the i-th database object in the j-th time slice, the k-th weighted feature is... Here, λ represents the original eigenvalues, and λ is the time decay factor. For the current time, A historical film, This is used to construct the state input tensor.

[0017] Furthermore, in step (2), the optimal transfer policy is generated through reinforcement learning, as shown in the following formula:

[0018] ;

[0019] in, For the optimal migration strategy, γ is the reward discount factor. For instant reward function, Resource consumption at time t This is the upper limit of the resource budget.

[0020] Furthermore, in step (3), a data migration mode is selected to generate a migration dataset, as shown in the following formula:

[0021] ;

[0022] in, To migrate the dataset, For the i-th data block, For the rate of change of data, The time window is the increment, and k is the initial number of objects to be migrated.

[0023] Furthermore, step (4) converts the source database SQL statement into the target SQL using a semantic mapping function, as shown in the following formula:

[0024] ;

[0025] in, For the i-th source SQL, For semantic mapping functions, SQL syntax diagram, Set a set of statements compatible with the target database.

[0026] Furthermore, step (5) performs a weighted integral verification on the target SQL transformed in step (4) and calculates the consistency score I, as follows:

[0027] ;

[0028] For source / destination hash, For indicator functions, Let z be the weighting function, z be the internal location of the data, I be the consistency score, and θ be the consistency judgment threshold. Statement set compatible with the target database The i-th data object being operated on; This refers to the migrated data in the target database.

[0029] Furthermore, step (6) writes the optimal migration strategy from step (2) and the consistency score I from step (5) into the blockchain, as shown in the following formula:

[0030] ;

[0031] in, For the k-th block, The hash of the previous block. For data summary, This is the current timestamp. The judgment result is obtained by calculating the consistency score I.

[0032] Furthermore, the instant reward function Defined as:

[0033] ;

[0034] in, The number of objects successfully migrated, The total number of objects, Used / maximum time For resource consumption and upper limit, is a coefficient.

[0035] A data migration device based on high-dimensional strategy optimization and end-to-end evidence storage includes:

[0036] Multidimensional tensor feature extraction module: used to collect dynamic features of the migration environment, such as resource utilization, data change frequency, and migration window, and generate a state input tensor through time decay weighted calculation;

[0037] Reinforcement policy generation module: used to generate the optimal transfer policy based on the state input tensor through reinforcement learning;

[0038] Migration dataset construction module: used to generate migration datasets by selecting a data migration mode according to the optimal migration strategy. The data migration mode includes full or incremental migration mode or a hybrid full or incremental migration mode.

[0039] SQL Semantic Mapping Transformation Module: Used to transform source database SQL statements into target database-compatible SQL using semantic mapping functions based on the output migration dataset;

[0040] Consistency integral verification module: used to perform weighted integral verification on the transformed target SQL and calculate the consistency score I;

[0041] Blockchain log storage module: Used to write the optimal migration strategy and consistency score I into the blockchain, build an auditable and tamper-proof migration chain, and complete the end-to-end storage.

[0042] A data migration system based on high-dimensional strategy optimization and end-to-end evidence storage includes: a computer-readable storage medium and a processor;

[0043] The computer-readable storage medium is used to store executable instructions;

[0044] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the data migration method based on high-dimensional strategy optimization and end-to-end evidence storage.

[0045] The present invention has the following advantages:

[0046] 1. High-dimensional modeling capability: By constructing time decay tensors, it comprehensively expresses the state evolution of database objects under multiple time slices and multiple indicators, effectively capturing the dynamic characteristics of the migration environment and providing accurate input for policy learning.

[0047] 2. Intelligent strategy optimization: By introducing reinforcement learning and Lagrange resource constraint mechanism, the optimal transfer strategy that takes into account success rate, resource consumption and time efficiency is dynamically generated, thereby improving the transfer adaptability and decision accuracy.

[0048] 3. Semantic-level SQL compatibility transformation: The syntax graph minimal mapping method is used to perform both structural and semantic transformations on SQL statements, ensuring logical consistency and execution effectiveness across heterogeneous database systems.

[0049] 4. Integral Consistency Verification: A weighted integral consistency function is designed to support dynamic evaluation of the consistency of migrated data according to the data importance distribution, thereby improving the ability to accurately judge the results of large-scale migrations.

[0050] 5. End-to-end traceability and evidence storage: Utilizing the blockchain structure to record strategies, summaries, and consistency verification results, a tamper-proof and auditable migration log chain is constructed to meet the compliance and traceability requirements of scenarios such as finance and government affairs.

[0051] 6. Tightly coupled modules and closed-loop logic: Each step is strictly connected to form a closed-loop architecture that covers the entire process from feature extraction, strategy decision-making, migration execution to consistency verification and on-chain evidence storage, resulting in strong system stability and feasibility. Attached Figure Description

[0052] Figure 1This is a flowchart of a data migration method based on high-dimensional strategy optimization and end-to-end evidence storage according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, the first aspect of this invention provides a data migration method based on high-dimensional strategy optimization and end-to-end evidence storage. Specifically, taking the migration from the source database system MySQL 5.7 to the target database system PostgreSQL 13 as an example, the entire process of this invention is demonstrated.

[0055] Step 1: Feature Tensor Construction

[0056] Data such as CPU utilization, write speed, and connection count are collected from the source database to form a state tensor. And generate a weighted tensor .

[0057] expression:

[0058]

[0059] Parameter settings:

[0060] λ = 0.1 (time decay factor)

[0061] Current time Time slice

[0062] Original eigenvalues

[0063] Example calculation:

[0064]

[0065] The sampling data is shown in Table 1:

[0066] Table 1

[0067]

[0068] Step 2: Reinforcement Strategy Generation

[0069] Generate transfer strategies using reinforcement learning models The goal is to maximize returns within the resource budget.

[0070] expression:

[0071]

[0072] Instant reward function:

[0073]

[0074] Parameter settings:

[0075]

[0076] Example calculation:

[0077]

[0078] Output strategy: Incremental migration ( =Increment)

[0079] Step 3: Migration Execution

[0080] According to strategy Select incremental migration to count the amount of data changes in the past 30 minutes.

[0081] The changed data is shown in Table 2:

[0082] Table 2

[0083]

[0084] Example calculation:

[0085]

[0086] Output: Incremental dataset size 43MB.

[0087] Step 4: SQL semantic mapping conversion

[0088] Convert the source database SQL statements into a format compatible with the target database.

[0089] Source SQL (MySQL):

[0090]

[0091] Target SQL (PostgreSQL):

[0092]

[0093] Mapping expression:

[0094]

[0095] Output: Transformed SQL set

[0096] Step 5: Consistency Integral Verification

[0097] A weighted integral is calculated to assess the consistency between source and target data.

[0098] Consistent expression:

[0099]

[0100] The input data (5 records) is shown in Table 3:

[0101] Table 3

[0102]

[0103] calculate:

[0104]

[0105] judge: The verification passed.

[0106] Output: Verification Report

[0107] Step 6: Blockchain Log Writing

[0108] Migration behavior is recorded in a chain structure.

[0109] Block expression:

[0110]

[0111] Example fields:

[0112]

[0113] Log chain structure:

[0114]

[0115] Based on the experimental results, the present invention has the following innovative features:

[0116] 1. Reinforcement Learning-Driven Policy Generation Mechanism: Traditional transfer learning methods rely on manually preset policies or fixed scheduling logic, lacking dynamic optimization capabilities. This invention constructs a reinforcement learning policy network with state tensors as input and introduces Lagrange multiplier constraints to achieve a multi-objective balance between resources, benefits, and timeliness, effectively improving the adaptability and success rate of transfer tasks under resource-constrained conditions.

[0117] 2. High-Dimensional Feature Tensor Modeling of Migration Environment State: Existing technologies generally use low-dimensional static indicators for policy judgment, which cannot accurately depict changes in the migration environment. This invention is the first to adopt a three-dimensional tensor (object × time × feature) structure and combine it with a time decay function to model the dynamic behavior during database migration, thereby improving the policy model's ability to perceive state changes.

[0118] 3. Integral-weighted Consistency Verification Model: Traditional consistency verification typically employs Boolean full-match mechanisms (such as CRC comparison), which cannot identify data importance or structural differences. This invention designs an integral-form consistency verification expression and introduces a weighting function. This enables a focus on evaluating the consistency of key data, significantly improving the accuracy of consistency judgment in scenarios involving incomplete data migration.

[0119] 4. A closed-loop end-to-end migration methodology: Existing migration systems generally suffer from fragmented modules and unverifiable processes. This invention tightly couples the "state modeling → policy generation → data construction → semantic transformation → consistency verification → evidence writing" stages through data flow, forming a logical closed loop and improving system stability and feasibility.

[0120] 5. Blockchain-based evidence storage mechanism ensures trustworthy data migration: This invention innovatively introduces a blockchain structure to record every migration behavior, strategy selection, and consistency verification result. By constructing a traceable full lifecycle log through a chain structure, it is suitable for scenarios with extremely high compliance requirements for migration behavior, such as finance, government affairs, and healthcare.

[0121] Another aspect of the present invention provides a data migration apparatus based on high-dimensional strategy optimization and end-to-end evidence storage, comprising:

[0122] Multidimensional tensor feature extraction module: used to collect dynamic features of the migration environment, such as resource utilization, data change frequency, and migration window, and generate a state input tensor through time decay weighted calculation;

[0123] Reinforcement policy generation module: used to generate the optimal transfer policy based on the state input tensor through reinforcement learning;

[0124] Migration dataset construction module: used to generate migration datasets by selecting a data migration mode according to the optimal migration strategy. The data migration mode includes full or incremental migration mode or a hybrid full or incremental migration mode.

[0125] SQL Semantic Mapping Transformation Module: Used to transform source database SQL statements into target database-compatible SQL using semantic mapping functions based on the output migration dataset;

[0126] Consistency integral verification module: used to perform weighted integral verification on the transformed target SQL and calculate the consistency score I;

[0127] Blockchain log storage module: Used to write the optimal migration strategy and consistency score I into the blockchain, constructing an auditable and tamper-proof migration chain, and completing end-to-end storage.

[0128] Another aspect of the present invention provides a data migration system based on high-dimensional strategy optimization and end-to-end evidence storage, comprising: a computer-readable storage medium and a processor;

[0129] The computer-readable storage medium is used to store executable instructions;

[0130] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the data migration method based on high-dimensional strategy optimization and end-to-end evidence storage as described in the first aspect.

[0131] In another aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data migration method based on high-dimensional strategy optimization and end-to-end evidence storage as described in the first aspect.

[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data migration method based on high-dimensional strategy optimization and end-to-end evidence storage, characterized in that, Includes the following steps: Step (1) Multidimensional tensor feature extraction: Collect dynamic features of the migration environment, such as resource utilization rate, data change frequency, and migration window, and generate a state input tensor by time decay weighting. Step (2) Reinforcement policy generation: Based on the state input tensor output in step (1), the optimal transfer policy is generated through reinforcement learning; Step (3) Construction of migration dataset: Based on the optimal migration strategy generated in step (2), select a data migration mode to generate a migration dataset. The data migration mode includes full or incremental migration mode or a hybrid migration mode of full or incremental. Step (4) SQL semantic mapping conversion: Based on the migration dataset output in step (3), the source database SQL statements are converted into target SQL in a format compatible with the target database through the semantic mapping function, so as to realize the equivalent conversion of SQL syntax between different databases and maintain the logical consistency before and after migration; Step (5) Consistency score verification: Perform a weighted integral verification on the target SQL transformed in step (4) and calculate the consistency score I; Step (6) Blockchain log storage: Write the optimal migration strategy from step (2) and the consistency score I from step (5) into the blockchain to build an auditable and tamper-proof migration chain and complete the full-link storage.

2. The method according to claim 1, characterized in that, In step (1), the state input tensor is generated by time decay weighting, as shown in the following formula: ； in, For the i-th database object in the j-th time slice, the k-th weighted feature is... Here, λ represents the original eigenvalues, and λ is the time decay factor. For the current time, A historical film, This is used to construct the state input tensor.

3. The method according to claim 1, characterized in that, In step (2), the optimal transfer policy is generated through reinforcement learning, as shown in the following formula: ； in, For the optimal migration strategy, γ is the reward discount factor. For instant reward function, Resource consumption at time t This is the upper limit of the resource budget.

4. The method according to claim 1, characterized in that, In step (3), a data migration mode is selected to generate a migration dataset, as shown in the following formula: ； in, To migrate the dataset, For the i-th data block, For the rate of change of data, The time window is the increment, and k is the initial number of objects to be migrated.

5. The method according to claim 1, characterized in that, Step (4) converts the source database SQL statement into the target SQL statement using a semantic mapping function, as shown in the following formula: ； in, For the i-th source SQL, For semantic mapping functions, SQL syntax diagram, Set a set of statements compatible with the target database.

6. The method according to claim 1, characterized in that, Step (5) performs a weighted integral verification on the target SQL transformed in step (4) and calculates the consistency score I, as follows: ； For source / destination hash, For indicator functions, Let z be the weighting function, z be the internal location of the data, I be the consistency score, and θ be the consistency judgment threshold. Statement set compatible with the target database The i-th data object being operated on; This refers to the migrated data in the target database.

7. The method according to claim 1, characterized in that, Step (6) writes the optimal migration strategy from step (2) and the consistency score I from step (5) into the blockchain, as shown in the following formula: ； in, For the k-th block, The hash of the previous block. For data summary, This is the current timestamp. The judgment result is obtained by calculating the consistency score I.

8. The method according to claim 3, characterized in that, The instant reward function Defined as: ； in, The number of objects successfully migrated, The total number of objects, Used / maximum time For resource consumption and upper limit, is a coefficient.

9. A data migration device based on high-dimensional strategy optimization and end-to-end evidence storage, characterized in that, include: Multidimensional tensor feature extraction module: used to collect dynamic features of the migration environment, such as resource utilization, data change frequency, and migration window, and generate a state input tensor through time decay weighted calculation; Reinforcement policy generation module: used to generate the optimal transfer policy based on the state input tensor through reinforcement learning; Migration dataset construction module: used to generate migration datasets by selecting a data migration mode according to the optimal migration strategy. The data migration mode includes full or incremental migration mode or a hybrid full or incremental migration mode. SQL Semantic Mapping Transformation Module: Used to transform source database SQL statements into target database-compatible SQL using semantic mapping functions based on the output migration dataset; Consistency integral verification module: used to perform weighted integral verification on the transformed target SQL and calculate the consistency score I; Blockchain log storage module: Used to write the optimal migration strategy and consistency score I into the blockchain, build an auditable and tamper-proof migration chain, and complete the end-to-end storage.

10. A data migration system based on high-dimensional strategy optimization and end-to-end evidence storage, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the data migration method based on high-dimensional strategy optimization and full-link evidence storage as described in any one of claims 1-8.

Images