AI-driven structured data migration method and system

Abstract

The application discloses an AI driving-based structured data migration method and system, belongs to the technical field of big data migration, and aims to solve the technical problem of how to improve the efficiency of large-scale data migration and reduce resource waste. The method comprises the following steps: forming a private large language model with field adaptation capability; performing lexical analysis and syntax analysis on non-standard structured data in a source database based on an AST analysis method, generating an abstract syntax tree AST, traversing and optimizing AST nodes, and converting the AST nodes into standard structured data supported by a target database; calling the large language model, and converting unstructured data into structured data supported by the target database through instruction fine-tuning, thought chain reasoning and step-by-step analysis; generating a migration script by the large language model; monitoring the migration process in real time, recording abnormal events, automatically retrying or repairing the abnormal events by the large model, and optimizing the large language model configuration strategy and data conversion rules in combination with feedback data.

Description

Technical Field

[0001] This invention relates to the field of big data migration technology, specifically to an AI-driven structured data migration method and system. Background Technology

[0002] Traditional data migration tools require engineers to program or configure them, which has the following disadvantages:

[0003] 1. Poor compatibility with heterogeneous systems: Syntax differences between different databases (such as Oracle and DM) lead to high migration costs and low efficiency of manual adaptation;

[0004] 2. Uncontrollable migration risks: Ensuring business continuity under complex dependencies is difficult and trial-and-error costs are high;

[0005] 3. Difficulty in processing unstructured data: Enterprise data contains unstructured content such as text and images, which traditional ETL tools struggle to handle efficiently.

[0006] The aforementioned drawbacks lead to problems such as low efficiency and serious resource waste during large-scale data migration. How to improve the efficiency of large-scale data migration and reduce resource waste is a technical problem that needs to be solved. Summary of the Invention

[0007] The technical objective of this invention is to address the above-mentioned shortcomings by providing an AI-driven structured data migration method and system to solve the technical problem of how to improve the efficiency of large-scale data migration and reduce resource waste.

[0008] In a first aspect, the present invention provides an AI-driven structured data migration method, comprising the following steps:

[0009] Private Large Language Model Configuration: Deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the anomaly handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities.

[0010] Non-standard structured data conversion: For non-standard structured data in the source database, lexical and syntactic analysis is performed based on the AST parsing method to generate an abstract syntax tree (AST). After traversing and optimizing the AST nodes, it is converted into standard structured data supported by the target database. After semantic verification and transaction isolation testing, standard structured data is output.

[0011] Unstructured data transformation: By calling a large language model, through instruction fine-tuning, thought chain reasoning and step-by-step parsing, regularized fields are extracted from unstructured data. After being filtered by a hybrid validation system, the extracted regularized fields are converted into structured data supported by the target database.

[0012] Data migration execution: For the generated structured data, a large language model generates a migration script, breaks down the migration task and dynamically schedules subtasks, and uses data splitting and parallel processing technology to load the converted structured data into the target database in chunks through synchronous or asynchronous transmission. Data integrity is ensured based on hash verification.

[0013] Anomaly Handling and Optimization: Monitor the migration process in real time, record abnormal events, automatically retry or repair them driven by the large model, continue the migration task from the point of failure, and optimize the large language model configuration strategy and data transformation rules based on feedback data.

[0014] As a preferred option, the configuration of a private large language model includes the following steps:

[0015] Local deployment: Deploy the open-source large language model and vector database locally and establish a connection;

[0016] Core parameter configuration: Extract dialect differences between the source and target databases to generate a data dialect configuration table. Divide the data scale levels based on the statistical information of the data to be migrated, configure the corresponding data processing strategies, define the anomaly types and handling rules during the data migration process, and form an anomaly handling rule set. Integrate the data dialect configuration table, data processing strategies, and anomaly handling rule set into an LLM parameter configuration file. The data scale levels are divided according to the number of tables and AST nodes. The processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and incremental synchronization in batches. The synchronization method supports single-threaded or cluster parallel processing. The anomaly types include data format errors, transmission interruptions, verification failures, and syntax incompatibility. The corresponding handling rules include LLM automatic correction, breakpoint resumption, automatic retransmission, and manual alarms. The anomaly log recording dimensions and monitoring alarm thresholds are also configured.

[0017] Knowledge base construction: Obtain domain-specific vocabulary mapping tables, historical data migration cases, and industry data conversion standards. After structuring, convert them into high-dimensional vectorized knowledge through an LLM embedding layer. Import the vectorized knowledge into a vector database and build a semantic index to construct a data conversion knowledge base that supports incremental updates. The semantic index is built based on the cosine similarity algorithm. The data conversion knowledge base is configured with an incremental update interface to support the import and index updates of newly added domain knowledge.

[0018] Domain-Adaptive LLM Generation: The LLM parameter configuration file is loaded into the locally deployed open-source large language model. Real-time linkage between the large language model and the data transformation knowledge base is established through a retrieval interface. Lightweight instructions are fine-tuned based on the data transformation knowledge base. After multi-scenario testing and iterative optimization, a private large language model with domain adaptation capabilities is generated. The lightweight instruction fine-tuning is based on historical migration cases in the data transformation knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, relevant domain knowledge is first obtained through the retrieval interface, and then a migration plan is generated in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data transformation knowledge base to achieve closed-loop iteration of knowledge.

[0019] As a preferred method, non-standard structured data transformation includes the following steps:

[0020] Lexical analysis: Obtain non-standard structured data, basic syntax rules, and non-standard format adaptation rules from the source database; construct a lexical analysis dictionary; extract and standardize lexical unit tokens; and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters, and constants. Standardization processing includes unifying keyword case, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value, and standardized value of each token.

[0021] Syntax Analysis: Based on the Token sequence, source database syntax rules, and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes, and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field lists and index definitions. The leaf nodes correspond to specific attributes, which include field names, data types, and constraints.

[0022] AST Node Traversal and Optimization: The visitor pattern is used to traverse the nodes at each level of the AST. Combining the target database syntax specifications, data type mapping rules, and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted, and redundancy is handled to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes, and supplementing missing required nodes.

[0023] Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the following: table structure definition statements, stored procedure code, and field definition list compatible with the target database. The data is formatted and standardized during the generation process.

[0024] Semantic validation and transaction isolation testing: Load the target database's semantic rules, business scenario transaction rules, and test cases. Perform semantic validity and structural integrity validation on the standard structured data. Simulate business concurrency scenarios to execute transaction isolation tests. After successful validation, output the final standard structured data. If the semantic validation or transaction isolation test fails, feedback is sent back to the corresponding preceding steps for adjustment until the test passes. Semantic validation includes field data type matching with business semantics, constraint validity validation, and logical consistency validation of associated fields. Transaction isolation testing includes monitoring the lock mechanism's running status, concurrency conflict detection, and data consistency validation. Transaction isolation levels include READCOMMITTED or REPEATABLE READ.

[0025] The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

[0026] As a preferred method, non-standard structured data transformation includes the following operations:

[0027] Instruction fine-tuning: A standardized domain-specific fine-tuning dataset is constructed based on the domain-specific fine-tuning dataset. Based on the standardized domain-specific fine-tuning dataset, task instruction templates, domain-specific vocabulary mapping tables, and instruction parameter configurations are extracted. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction includes unstructured data input, domain field extraction requirements, and expected output. After fine-tuning and validating the preliminary domain-adapted large language model, a domain-adapted large language model is generated.

[0028] Mind Chain Reasoning: Input the unstructured data to be processed and the mind chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction. Output a preliminary set of regularized fields and mind chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files.

[0029] Step-by-step analysis: Split the unstructured data to be processed according to the data segmentation rules, perform mind chain reasoning to extract rule-based fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule-based fields;

[0030] Hybrid verification system filtering: Based on confidence threshold, field verification rule base and manual review triggering conditions, the precise rule-based field set is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final valid rule-based field set;

[0031] Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final valid rule-based field set is converted into standard structured data supported by the target database. If subsequent steps discover field quality, completeness or compatibility issues, the feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met.

[0032] Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

[0033] As a preferred method, the data migration process includes the following operations:

[0034] Migration script generation: Based on the transformed standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization;

[0035] Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules, and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified.

[0036] Subtask dynamic scheduling: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted.

[0037] Data splitting and parallel processing: Standardized structured data is split into shards according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data shards with source hash values;

[0038] Data fragment transmission: Based on the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database synchronously or asynchronously through an encrypted transmission channel, and the transmission status is recorded;

[0039] Data sharding loading: Based on the target database table structure information and data loading rules, the data transmitted to the target node is sharded and loaded into the target database in batches, supporting transaction isolation and failure retry;

[0040] Data integrity verification: Based on the hash algorithm configuration, calculate the hash value of the data shards at the source and destination ends respectively, and verify data integrity by comparing the shard level and the full summary level, and handle inconsistent data;

[0041] Among them, the migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data shard transmission, data shard loading, and data integrity verification are executed, and corresponding log files are generated. These are the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data shard transmission log, and data integrity verification log.

[0042] Secondly, the present invention provides an AI-driven structured data migration system, comprising a private large language model configuration module, a non-standard structured data conversion module, an unstructured data conversion module, a data migration execution module, and an exception handling and optimization module;

[0043] The private large language model configuration module is used to perform the following: deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the exception handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities.

[0044] The non-standard structured data conversion module performs the following: For non-standard structured data in the source database, it performs lexical and syntactic analysis based on the AST parsing method, generates an abstract syntax tree (AST), traverses and optimizes the AST nodes, converts it into standard structured data supported by the target database, and outputs standard structured data after semantic verification and transaction isolation testing.

[0045] The unstructured data transformation module is used to perform the following: call the large language model, extract regularized fields from the unstructured data through instruction fine-tuning, thought chain reasoning and step-by-step parsing, and after filtering by the hybrid validation system, convert the extracted regularized fields into structured data supported by the target database;

[0046] The data migration execution module is used to perform the following: For the generated structured data, a migration script is generated by the large language model, the migration task is broken down and sub-tasks are dynamically scheduled, data splitting and parallel processing technology is used, and the converted structured data is loaded into the target database in chunks through synchronous or asynchronous transmission, and data integrity is ensured based on hash verification.

[0047] The exception handling and optimization module is used to perform the following: real-time monitoring of the migration process, recording of abnormal events, automatic retrying or large model-driven repair, continuing the migration task from the point of failure, and optimizing the large language model configuration strategy and data transformation rules based on feedback data.

[0048] As a preferred option, the private large language model configuration module is used to perform the following operations:

[0049] Local deployment: Deploy the open-source large language model and vector database locally and establish a connection;

[0050] Core parameter configuration: Extract dialect differences between the source and target databases to generate a data dialect configuration table. Divide the data scale levels based on the statistical information of the data to be migrated, configure the corresponding data processing strategies, define the anomaly types and handling rules during the data migration process, and form an anomaly handling rule set. Integrate the data dialect configuration table, data processing strategies, and anomaly handling rule set into an LLM parameter configuration file. The data scale levels are divided according to the number of tables and AST nodes. The processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and incremental synchronization in batches. The synchronization method supports single-threaded or cluster parallel processing. The anomaly types include data format errors, transmission interruptions, verification failures, and syntax incompatibility. The corresponding handling rules include LLM automatic correction, breakpoint resumption, automatic retransmission, and manual alarms. The anomaly log recording dimensions and monitoring alarm thresholds are also configured.

[0051] Knowledge base construction: Obtain domain-specific vocabulary mapping tables, historical data migration cases, and industry data conversion standards. After structuring, convert them into high-dimensional vectorized knowledge through an LLM embedding layer. Import the vectorized knowledge into a vector database and build a semantic index to construct a data conversion knowledge base that supports incremental updates. The semantic index is built based on the cosine similarity algorithm. The data conversion knowledge base is configured with an incremental update interface to support the import and index updates of newly added domain knowledge.

[0052] Domain-Adaptive LLM Generation: The LLM parameter configuration file is loaded into the locally deployed open-source large language model. Real-time linkage between the large language model and the data transformation knowledge base is established through a retrieval interface. Lightweight instructions are fine-tuned based on the data transformation knowledge base. After multi-scenario testing and iterative optimization, a private large language model with domain adaptation capabilities is generated. The lightweight instruction fine-tuning is based on historical migration cases in the data transformation knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, relevant domain knowledge is first obtained through the retrieval interface, and then a migration plan is generated in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data transformation knowledge base to achieve closed-loop iteration of knowledge.

[0053] Preferably, the non-standard structured data conversion module is used to perform the following operations:

[0054] Lexical analysis: Obtain non-standard structured data, basic syntax rules, and non-standard format adaptation rules from the source database; construct a lexical analysis dictionary; extract and standardize lexical unit tokens; and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters, and constants. Standardization processing includes unifying keyword case, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value, and standardized value of each token.

[0055] Syntax Analysis: Based on the Token sequence, source database syntax rules, and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes, and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field lists and index definitions. The leaf nodes correspond to specific attributes, which include field names, data types, and constraints.

[0056] AST Node Traversal and Optimization: The visitor pattern is used to traverse the nodes at each level of the AST. Combining the target database syntax specifications, data type mapping rules, and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted, and redundancy is handled to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes, and supplementing missing required nodes.

[0057] Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the following: table structure definition statements, stored procedure code, and field definition list compatible with the target database. The data is formatted and standardized during the generation process.

[0058] Semantic validation and transaction isolation testing: Load the target database's semantic rules, business scenario transaction rules, and test cases. Perform semantic validity and structural integrity validation on the standard structured data. Simulate business concurrency scenarios to execute transaction isolation tests. After successful validation, output the final standard structured data. If the semantic validation or transaction isolation test fails, feedback is sent back to the corresponding preceding steps for adjustment until the test passes. Semantic validation includes field data type matching with business semantics, constraint validity validation, and logical consistency validation of associated fields. Transaction isolation testing includes monitoring the lock mechanism's running status, concurrency conflict detection, and data consistency validation. Transaction isolation levels include READCOMMITTED or REPEATABLE READ.

[0059] The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

[0060] Preferably, the non-standard structured data conversion module is used to perform the following operations:

[0061] Instruction fine-tuning: A standardized domain-specific fine-tuning dataset is constructed based on the domain-specific fine-tuning dataset. Based on the standardized domain-specific fine-tuning dataset, task instruction templates, domain-specific vocabulary mapping tables, and instruction parameter configurations are extracted. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction includes unstructured data input, domain field extraction requirements, and expected output. After fine-tuning and validating the preliminary domain-adapted large language model, a domain-adapted large language model is generated.

[0062] Mind Chain Reasoning: Input the unstructured data to be processed and the mind chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction. Output a preliminary set of regularized fields and mind chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files.

[0063] Step-by-step analysis: Split the unstructured data to be processed according to the data segmentation rules, perform mind chain reasoning to extract rule-based fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule-based fields;

[0064] Hybrid verification system filtering: Based on confidence threshold, field verification rule base and manual review triggering conditions, the precise rule-based field set is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final valid rule-based field set;

[0065] Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final valid rule-based field set is converted into standard structured data supported by the target database. If subsequent steps discover field quality, completeness or compatibility issues, the feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met.

[0066] Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

[0067] Preferably, the data migration execution module is used to perform the following operations:

[0068] Migration script generation: Based on the transformed standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization;

[0069] Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules, and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified.

[0070] Subtask dynamic scheduling: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted.

[0071] Data splitting and parallel processing: Standardized structured data is split into shards according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data shards with source hash values;

[0072] Data fragment transmission: Based on the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database synchronously or asynchronously through an encrypted transmission channel, and the transmission status is recorded;

[0073] Data sharding loading: Based on the target database table structure information and data loading rules, the data transmitted to the target node is sharded and loaded into the target database in batches, supporting transaction isolation and failure retry;

[0074] Data integrity verification: Based on the hash algorithm configuration, calculate the hash value of the data shards at the source and destination ends respectively, and verify data integrity by comparing the shard level and the full summary level, and handle inconsistent data;

[0075] Among them, the migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data shard transmission, data shard loading, and data integrity verification are executed, and corresponding log files are generated. These are the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data shard transmission log, and data integrity verification log.

[0076] The AI-driven structured data migration method and system of the present invention have the following advantages:

[0077] 1. Automation and Efficiency Improvement: Automated code migration; large models can be automatically parsed and transformed into code logic, significantly reducing manual workload;

[0078] 2. Structured and unstructured data conversion capability: Large models can extract regularized fields (such as tables and JSON) from unstructured data such as text and images, achieving multimodal alignment of unstructured data to structured data;

[0079] 3. Enhanced Quality Assurance and Security: Iterative optimization driven by large model validation requires automated validation for each migration step. Failure triggers an LLM repair loop. By recording migration status and targeted retries, the data conversion success rate has increased from 75% to 97%. Data conversion can be performed locally, with only anonymized information transmitted to the cloud, thus avoiding data security issues. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] The invention will be further described below with reference to the accompanying drawings.

[0082] Figure 1 This is a flowchart of an AI-driven structured data migration method according to Example 1. Detailed Implementation

[0083] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0084] This invention provides an AI-driven structured data migration method and system to address the technical problem of how to improve the efficiency of large-scale data migration and reduce resource waste.

[0085] Example 1:

[0086] This invention provides an AI-driven structured data migration method, comprising five steps: private large language model configuration, non-standard structured data conversion, unstructured data conversion, data migration execution, and anomaly handling and optimization.

[0087] Step S100: Configure the private large language model: Deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the exception handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities.

[0088] As a specific implementation of the private large language model configuration, this step includes the following operations:

[0089] (1) Local deployment: Deploy the open-source large language model and vector database locally and establish a connection;

[0090] (2) Core parameter configuration: Extract the dialect differences between the source database and the target database, generate a data dialect configuration table, divide the data scale level based on the statistical information of the data to be migrated, configure the corresponding data processing strategy, define the anomaly types and processing rules in the data migration process, form an anomaly processing rule set, integrate the data dialect configuration table, data processing strategy and anomaly processing rule set into an LLM parameter configuration file, wherein the data scale level is divided according to the number of tables and the number of AST nodes, and the processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and batch incremental synchronization, and the synchronization method supports single-thread or cluster parallel processing, the anomaly types include data format error, transmission interruption, verification failure and syntax incompatibility, and the corresponding processing rules include LLM automatic correction, breakpoint resume, automatic retransmission and manual alarm, and configure the anomaly log recording dimension and monitoring alarm threshold;

[0091] (3) Knowledge base construction: Obtain domain-specific vocabulary mapping table, historical data migration cases and industry data conversion standards, perform structured processing and convert them into high-dimensional vectorized knowledge through LLM embedding layer, import the vectorized knowledge into vector database and establish semantic index, and build a data conversion knowledge base that supports incremental updates. The semantic index is built based on cosine similarity algorithm, and the data conversion knowledge base is configured with incremental update interface to support the import and index update of newly added domain knowledge.

[0092] (4) Domain-Adaptive LLM Generation: Load the LLM parameter configuration file into the locally deployed open-source large language model, establish real-time linkage between the large language model and the data conversion knowledge base through the retrieval interface, perform lightweight instruction fine-tuning based on the data conversion knowledge base, and generate a private large language model with domain adaptation capabilities after multi-scenario testing and iterative optimization. The lightweight instruction fine-tuning is based on historical migration cases in the data conversion knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, first obtain relevant domain knowledge through the retrieval interface, and then generate a migration plan in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data conversion knowledge base to achieve knowledge closed-loop iteration.

[0093] Step S200 Non-standard structured data conversion: For non-standard structured data in the source database, lexical and syntactic analysis is performed based on the AST parsing method to generate an abstract syntax tree (AST). After traversing and optimizing the AST nodes, it is converted into standard structured data supported by the target database. After semantic verification and transaction isolation testing, standard structured data is output.

[0094] As a specific implementation of non-standard structured data transformation, this step includes the following operations:

[0095] (1) Lexical analysis: Obtain non-standard structured data, basic syntax rules and non-standard format adaptation rules of the source database, construct a lexical analysis dictionary, extract and standardize lexical unit tokens, and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters and constants. The standardization process includes unifying the case of keywords, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value and standardized value of each token.

[0096] (2) Syntax analysis: Based on the Token sequence, source database syntax rules and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field list and index definition. The leaf nodes correspond to specific attributes, which include field name, data type and constraint conditions.

[0097] (3) AST node traversal and optimization: The visitor pattern is used to traverse the nodes of the AST at all levels. Combined with the target database syntax specifications, data type mapping rules and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted and redundancy processed to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes and supplementing missing required nodes.

[0098] (4) Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the table structure definition statements, stored procedure code and field definition list that are compatible with the target database. The data is formatted and standardized during the generation process.

[0099] (5) Semantic validation and transaction isolation test: Load the target database semantic rules, business scenario transaction rules and test cases, perform semantic legality and structural integrity validation on the standard structured data, simulate business concurrency scenarios to perform transaction isolation test, and output the final standard structured data after the validation is passed. If the semantic validation or transaction isolation test fails, it is fed back to the corresponding preceding steps for adjustment until the test is passed. Semantic validation includes field data type matching with business semantics, constraint legality validation, and logical consistency validation of related fields. Transaction isolation test includes lock mechanism operation status monitoring, concurrency conflict detection, and data consistency validation. Transaction isolation level includes READCOMMITTED or REPEATABLE READ.

[0100] The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

[0101] Step S300 Unstructured Data Transformation: The large language model is invoked, and through instruction fine-tuning, thought chain reasoning, and step-by-step parsing, regularized fields are extracted from the unstructured data. After filtering by the hybrid validation system, the extracted regularized fields are converted into structured data supported by the target database.

[0102] As a specific implementation of non-standard structured data transformation, this step includes the following operations:

[0103] (1) Instruction fine-tuning: Based on the domain-specific fine-tuning dataset, a standardized domain-specific fine-tuning dataset is constructed. Based on the standardized domain-specific fine-tuning dataset, task instruction templates are extracted, domain-specific vocabulary mapping tables and instruction parameter configurations are configured. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction contains unstructured data input, domain field extraction requirements and expected output. After fine-tuning the preliminary domain-adapted large language model and verifying its effects, a domain-adapted large language model is generated.

[0104] (2) Thinking Chain Reasoning: Input the unstructured data to be processed and the thinking chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction, and output the preliminary rule-based field set and thinking chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files.

[0105] (3) Step-by-step analysis: Split the unstructured data to be processed according to the data block rules, perform mind chain reasoning to extract the rule fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule fields;

[0106] (4) Hybrid verification system filtering: Based on the confidence threshold, field verification rule base and manual review triggering conditions, the set of precise rule-based fields is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final set of valid rule-based fields;

[0107] (5) Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final effective set of regularized fields is converted into standard structured data supported by the target database. If subsequent steps discover field quality, integrity or compatibility issues, feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met.

[0108] Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

[0109] Step S400 Data Migration Execution: For the generated structured data, a migration script is generated by the large language model, the migration task is broken down and sub-tasks are dynamically scheduled, and data splitting and parallel processing technology is adopted. The converted structured data is loaded into the target database in chunks through synchronous or asynchronous transmission, and data integrity is ensured based on hash verification.

[0110] As a specific implementation of the data migration process, this step includes the following operations:

[0111] (1) Migration script generation: Based on the converted standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization;

[0112] (2) Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified;

[0113] (3) Dynamic scheduling of subtasks: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, the execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted.

[0114] (4) Data splitting and parallel processing: Standardized structured data is split into pieces according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data pieces with source hash values;

[0115] (5) Data fragment transmission: According to the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database in a synchronous or asynchronous manner through an encrypted transmission channel, and the transmission status is recorded;

[0116] (6) Data shard loading: According to the target database table structure information and data loading rules, the data shards transmitted to the target node are loaded into the target database in batches, supporting transaction isolation and failure retry;

[0117] (7) Data integrity verification: According to the hash algorithm configuration, calculate the hash value of the data fragments at the source and destination ends respectively, and verify the data integrity by comparing the fragment level and the full summary level, and handle inconsistent data;

[0118] Among them, the migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data shard transmission, data shard loading, and data integrity verification are executed, and corresponding log files are generated. These are the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data shard transmission log, and data integrity verification log.

[0119] In this process, for each execution step in the migration script, compatibility score, transformation cost, and business continuity indicators are defined as states, and each execution step is defined as an action. A Markov decision process is constructed, and the decision strategy is iteratively updated using a Q-learning algorithm to generate a global migration path planning scheme. A domain adaptation algorithm is used to align data features between the source and target databases. Data features include data structure, data distribution, and syntax rules. For each execution step in the execution script, path planning is performed using Markov decision-making. Specifically, the state space includes system compatibility score, transformation cost, and business continuity indicators; the action space includes the migration step sequence (e.g., data cleaning → syntax conversion → verification testing); and the reward function is defined as follows:

[0120] R(s,a) = α(1-risk coefficient) + βefficiency improvement rate

[0121] Update the strategy using the Q-learning algorithm:

[0122] Q(s_t,a_t) ← Q(s_t,a_t) + α[r_{t+1} + γ\max_a Q(s_{t+1},a) - Q(s_t,a_t)].

[0123] The domain adaptation algorithm is used to represent this as follows:

[0124] L = L_{task} + λL_{MMD},

[0125] LMMD, or Maximum Mean Difference Loss, is used to align the feature distributions of the source and target domains.

[0126] Step S500: Anomaly Handling and Optimization: Monitor the migration process in real time, record abnormal events, automatically retry or repair them using the large model, continue the migration task from the point of failure, and optimize the large language model configuration strategy and data transformation rules based on feedback data.

[0127] The method in this embodiment utilizes a Large Language Model (LLM) to automatically analyze legacy system code and generate adaptable code, reducing manual intervention and significantly improving migration efficiency. It can replace manual experience and reduce the workload of data migration. Suppose we need to migrate the schema of an Oracle database to a new version of PostgreSQL, where there are differences in data types, index syntax, etc. Traditional regular expression replacement is prone to errors, while AST parsing can accurately handle complex structures. The method in this embodiment has the following functions:

[0128] 1. Automated Data Integration: One of its biggest features is its high degree of automation. It can automatically detect changes in the data structure of the data source and automatically configure data synchronization tasks, greatly reducing the workload of manually configuring and maintaining data pipelines. Users only need to provide the connection information of the data source and the information of the target warehouse, and the data extraction, transformation, and loading (ETL) process can be completed automatically.

[0129] 2. Broad Data Source Support: Supports nearly a hundred different data sources, including various popular SaaS applications, databases (relational and non-relational databases), file storage systems, etc. This allows enterprises to easily integrate data from different systems into a unified data warehouse for comprehensive analysis.

[0130] 3. Real-time and near real-time data synchronization: Provides real-time or near real-time data synchronization capabilities, ensuring that data in the data warehouse is always up-to-date. This feature is crucial for enterprises that need to make timely decisions, enabling them to analyze and make decisions based on the latest data.

[0131] 4. Low-code or no-code operation: Data integration tasks can be completed without writing large amounts of code. Its user interface is simple and intuitive, making it easy for non-technical personnel to get started, lowering the technical threshold for data integration and improving the collaboration efficiency between different departments within an enterprise.

[0132] 5. Data Quality and Monitoring: Provides data quality monitoring and management functions, capable of automatically detecting errors, missing values, and other issues in the data, and providing corresponding reports and alerts. Simultaneously, users can monitor the data integration process, gaining real-time insight into the data synchronization status and performance.

[0133] 6. Improved migration efficiency: Resolves performance bottlenecks in heterogeneous database migration by shortening migration time through parallel processing; enables automatic adaptation of metadata storage structures for multiple databases, reducing manual intervention; optimizes the data conversion process, reducing resource consumption and computational costs during migration.

[0134] Example 2:

[0135] This invention discloses an AI-driven structured data migration system, comprising a private large language model configuration module, a non-standard structured data conversion module, an unstructured data conversion module, a data migration execution module, and an exception handling and optimization module.

[0136] The private large language model configuration module is used to perform the following: deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the exception handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities.

[0137] As a specific implementation of the private large language model configuration module, this module is used to perform the following operations:

[0138] (1) Local deployment: Deploy the open-source large language model and vector database locally and establish a connection;

[0139] (2) Core parameter configuration: Extract the dialect differences between the source database and the target database, generate a data dialect configuration table, divide the data scale level based on the statistical information of the data to be migrated, configure the corresponding data processing strategy, define the anomaly types and processing rules in the data migration process, form an anomaly processing rule set, integrate the data dialect configuration table, data processing strategy and anomaly processing rule set into an LLM parameter configuration file, wherein the data scale level is divided according to the number of tables and the number of AST nodes, and the processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and batch incremental synchronization, and the synchronization method supports single-thread or cluster parallel processing, the anomaly types include data format error, transmission interruption, verification failure and syntax incompatibility, and the corresponding processing rules include LLM automatic correction, breakpoint resume, automatic retransmission and manual alarm, and configure the anomaly log recording dimension and monitoring alarm threshold;

[0140] (3) Knowledge base construction: Obtain domain-specific vocabulary mapping table, historical data migration cases and industry data conversion standards, perform structured processing and convert them into high-dimensional vectorized knowledge through LLM embedding layer, import the vectorized knowledge into vector database and establish semantic index, and build a data conversion knowledge base that supports incremental updates. The semantic index is built based on cosine similarity algorithm, and the data conversion knowledge base is configured with incremental update interface to support the import and index update of newly added domain knowledge.

[0141] (4) Domain-Adaptive LLM Generation: Load the LLM parameter configuration file into the locally deployed open-source large language model, establish real-time linkage between the large language model and the data conversion knowledge base through the retrieval interface, perform lightweight instruction fine-tuning based on the data conversion knowledge base, and generate a private large language model with domain adaptation capabilities after multi-scenario testing and iterative optimization. The lightweight instruction fine-tuning is based on historical migration cases in the data conversion knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, first obtain relevant domain knowledge through the retrieval interface, and then generate a migration plan in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data conversion knowledge base to achieve knowledge closed-loop iteration.

[0142] The non-standard structured data conversion module performs the following: For non-standard structured data in the source database, it performs lexical and syntactic analysis based on the AST parsing method, generates an abstract syntax tree (AST), traverses and optimizes the AST nodes, converts it into standard structured data supported by the target database, and outputs standard structured data after semantic verification and transaction isolation testing.

[0143] As a specific implementation of the non-standard structured data conversion module, this module is used to perform the following operations:

[0144] (1) Lexical analysis: Obtain non-standard structured data, basic syntax rules and non-standard format adaptation rules of the source database, construct a lexical analysis dictionary, extract and standardize lexical unit tokens, and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters and constants. The standardization process includes unifying the case of keywords, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value and standardized value of each token.

[0145] (2) Syntax analysis: Based on the Token sequence, source database syntax rules and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field list and index definition. The leaf nodes correspond to specific attributes, which include field name, data type and constraint conditions.

[0146] (3) AST node traversal and optimization: The visitor pattern is used to traverse the nodes of the AST at all levels. Combined with the target database syntax specifications, data type mapping rules and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted and redundancy processed to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes and supplementing missing required nodes.

[0147] (4) Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the table structure definition statements, stored procedure code and field definition list that are compatible with the target database. The data is formatted and standardized during the generation process.

[0148] (5) Semantic validation and transaction isolation test: Load the target database semantic rules, business scenario transaction rules and test cases, perform semantic legality and structural integrity validation on the standard structured data, simulate business concurrency scenarios to perform transaction isolation test, and output the final standard structured data after the validation is passed. If the semantic validation or transaction isolation test fails, it is fed back to the corresponding preceding steps for adjustment until the test is passed. Semantic validation includes field data type matching with business semantics, constraint legality validation, and logical consistency validation of related fields. Transaction isolation test includes lock mechanism operation status monitoring, concurrency conflict detection, and data consistency validation. Transaction isolation level includes READCOMMITTED or REPEATABLE READ.

[0149] The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

[0150] The unstructured data transformation module is used to perform the following: call the large language model, extract regularized fields from the unstructured data through instruction fine-tuning, thought chain reasoning and step-by-step parsing, and after filtering by the hybrid validation system, convert the extracted regularized fields into structured data supported by the target database.

[0151] As a specific implementation of the non-standard structured data conversion module, this module is used to perform the following operations:

[0152] (1) Instruction fine-tuning: Based on the domain-specific fine-tuning dataset, a standardized domain-specific fine-tuning dataset is constructed. Based on the standardized domain-specific fine-tuning dataset, task instruction templates are extracted, domain-specific vocabulary mapping tables and instruction parameter configurations are configured. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction contains unstructured data input, domain field extraction requirements and expected output. After fine-tuning the preliminary domain-adapted large language model and verifying its effects, a domain-adapted large language model is generated.

[0153] (2) Thinking Chain Reasoning: Input the unstructured data to be processed and the thinking chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction, and output the preliminary rule-based field set and thinking chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files.

[0154] (3) Step-by-step analysis: Split the unstructured data to be processed according to the data block rules, perform mind chain reasoning to extract the rule fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule fields;

[0155] (4) Hybrid verification system filtering: Based on the confidence threshold, field verification rule base and manual review triggering conditions, the set of precise rule-based fields is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final set of valid rule-based fields;

[0156] (5) Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final effective set of regularized fields is converted into standard structured data supported by the target database. If subsequent steps discover field quality, integrity or compatibility issues, feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met.

[0157] Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

[0158] The data migration execution module performs the following: For the generated structured data, a large language model generates a migration script, breaks down the migration task and dynamically schedules subtasks, uses data splitting and parallel processing techniques, and loads the converted structured data into the target database in chunks through synchronous or asynchronous transmission. Data integrity is ensured based on hash verification. For the split migration task, the migration task is taken as the action, and compatibility score, transformation cost and business continuity are taken as the states. The migration path is planned through Erkov decision, and the data feature alignment between the source database and the target database is achieved through a domain adaptive algorithm. The data features include data structure, data distribution and syntax rules.

[0159] As a specific implementation of the data migration execution module, this module is used to perform the following operations:

[0160] (1) Migration script generation: Based on the converted standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization;

[0161] (2) Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified;

[0162] (3) Dynamic scheduling of subtasks: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, the execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted.

[0163] (4) Data splitting and parallel processing: Standardized structured data is split into pieces according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data pieces with source hash values;

[0164] (5) Data fragment transmission: According to the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database in a synchronous or asynchronous manner through an encrypted transmission channel, and the transmission status is recorded;

[0165] (6) Data shard loading: According to the target database table structure information and data loading rules, the data shards transmitted to the target node are loaded into the target database in batches, supporting transaction isolation and failure retry;

[0166] (7) Data integrity verification: According to the hash algorithm configuration, calculate the hash value of the data fragments at the source and destination ends respectively, and verify the data integrity by comparing the fragment level and the full summary level, and handle inconsistent data;

[0167] Among them, the migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data shard transmission, data shard loading, and data integrity verification are executed, and corresponding log files are generated. These are the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data shard transmission log, and data integrity verification log.

[0168] In this process, for each execution step in the migration script, compatibility score, transformation cost, and business continuity indicators are defined as states, and each execution step is defined as an action. A Markov decision process is constructed, and the decision strategy is iteratively updated using a Q-learning algorithm to generate a global migration path planning scheme. A domain adaptation algorithm is used to align data features between the source and target databases. Data features include data structure, data distribution, and syntax rules. For each execution step in the execution script, path planning is performed using Markov decision-making. Specifically, the state space includes system compatibility score, transformation cost, and business continuity indicators; the action space includes the migration step sequence (e.g., data cleaning → syntax conversion → verification testing); and the reward function is defined as follows:

[0169] R(s,a) = α(1-risk coefficient) + βefficiency improvement rate

[0170] Update the strategy using the Q-learning algorithm:

[0171] Q(s_t,a_t) ← Q(s_t,a_t) + α[r_{t+1} + γ\max_a Q(s_{t+1},a) - Q(s_t,a_t)].

[0172] The domain adaptation algorithm is used to represent this as follows:

[0173] L = L_{task} + λL_{MMD},

[0174] LMMD, or Maximum Mean Difference Loss, is used to align the feature distributions of the source and target domains.

[0175] The exception handling and optimization module is used to perform the following: real-time monitoring of the migration process, recording of abnormal events, automatic retrying or large model-driven repair, continuing the migration task from the point of failure, and optimizing the large language model configuration strategy and data transformation rules based on feedback data.

[0176] The system in this embodiment can execute the method disclosed in Embodiment 1 to realize database migration, especially in scenarios where multiple databases coexist and data storage methods are relatively complex. It is suitable for migration and conversion between different databases with complex data structures, and is suitable for heterogeneous data integration in industries such as finance and telecommunications.

[0177] The AI-driven structured data migration method and system provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An AI-driven structured data migration method, characterized by, Includes the following steps: Private Large Language Model Configuration: Deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the anomaly handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities. Non-standard structured data conversion: For non-standard structured data in the source database, lexical and syntactic analysis is performed based on the AST parsing method to generate an abstract syntax tree (AST). After traversing and optimizing the AST nodes, it is converted into standard structured data supported by the target database. After semantic verification and transaction isolation testing, standard structured data is output. Unstructured data transformation: By calling a large language model, through instruction fine-tuning, thought chain reasoning and step-by-step parsing, regularized fields are extracted from unstructured data. After being filtered by a hybrid validation system, the extracted regularized fields are converted into structured data supported by the target database. Data migration execution: For the generated structured data, a large language model generates migration scripts, breaks down migration tasks and dynamically schedules subtasks, and uses data splitting and parallel processing techniques to load the transformed structured data into the target database in chunks through synchronous or asynchronous transmission. Data integrity is ensured based on hash verification. For the split migration tasks, the migration tasks are used as actions, and compatibility scores, transformation costs, and business continuity are used as states. The migration path is planned through Erkov decision-making, and the data features of the source database and the target database are aligned through a domain adaptation algorithm. The data features include data structure, data distribution, and syntax rules. The migration tasks include data cleaning, syntax conversion, verification testing, data sharding, parallel transmission, loading verification, and anomaly repair. Anomaly Handling and Optimization: Monitor the migration process in real time, record abnormal events, automatically retry or repair them driven by the large model, continue the migration task from the point of failure, and optimize the large language model configuration strategy and data transformation rules based on feedback data.

2. The Al-driven structured data migration method of claim 1, wherein, Configuring a private large language model includes the following steps: Local deployment: Deploy the open-source large language model and vector database locally and establish a connection; Core parameter configuration: Extract dialect differences between the source and target databases to generate a data dialect configuration table. Divide the data scale levels based on the statistical information of the data to be migrated, configure the corresponding data processing strategies, define the anomaly types and handling rules during the data migration process, and form an anomaly handling rule set. Integrate the data dialect configuration table, data processing strategies, and anomaly handling rule set into an LLM parameter configuration file. The data scale levels are divided according to the number of tables and AST nodes. The processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and incremental synchronization in batches. The synchronization method supports single-threaded or cluster parallel processing. The anomaly types include data format errors, transmission interruptions, verification failures, and syntax incompatibility. The corresponding handling rules include LLM automatic correction, breakpoint resumption, automatic retransmission, and manual alarms. The anomaly log recording dimensions and monitoring alarm thresholds are also configured. Knowledge base construction: Obtain domain-specific vocabulary mapping tables, historical data migration cases, and industry data conversion standards. After structuring, convert them into high-dimensional vectorized knowledge through an LLM embedding layer. Import the vectorized knowledge into a vector database and build a semantic index to construct a data conversion knowledge base that supports incremental updates. The semantic index is built based on the cosine similarity algorithm. The data conversion knowledge base is configured with an incremental update interface to support the import and index updates of newly added domain knowledge. Domain-Adaptive LLM Generation: The LLM parameter configuration file is loaded into the locally deployed open-source large language model. Real-time linkage between the large language model and the data transformation knowledge base is established through a retrieval interface. Lightweight instructions are fine-tuned based on the data transformation knowledge base. After multi-scenario testing and iterative optimization, a private large language model with domain adaptation capabilities is generated. The lightweight instruction fine-tuning is based on historical migration cases in the data transformation knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, relevant domain knowledge is first obtained through the retrieval interface, and then a migration plan is generated in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data transformation knowledge base to achieve closed-loop iteration of knowledge.

3. The Al-driven structured data migration method of claim 1, wherein, Non-standard structured data transformation includes the following steps: Lexical analysis: Obtain non-standard structured data, basic syntax rules, and non-standard format adaptation rules from the source database; construct a lexical analysis dictionary; extract and standardize lexical unit tokens; and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters, and constants. Standardization processing includes unifying keyword case, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value, and standardized value of each token. Syntax Analysis: Based on the Token sequence, source database syntax rules, and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes, and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field lists and index definitions. The leaf nodes correspond to specific attributes, which include field names, data types, and constraints. AST Node Traversal and Optimization: The visitor pattern is used to traverse the nodes at each level of the AST. Combining the target database syntax specifications, data type mapping rules, and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted, and redundancy is handled to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes, and supplementing missing required nodes. Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the following: table structure definition statements, stored procedure code, and field definition list compatible with the target database. The data is formatted and standardized during the generation process. Semantic validation and transaction isolation testing: Load the target database's semantic rules, business scenario transaction rules, and test cases. Perform semantic validity and structural integrity validation on the standard structured data. Simulate business concurrency scenarios to execute transaction isolation tests. After successful validation, output the final standard structured data. If the semantic validation or transaction isolation test fails, feedback is sent back to the corresponding preceding steps for adjustment until the test passes. Semantic validation includes field data type matching with business semantics, constraint validity validation, and logical consistency validation of associated fields. Transaction isolation testing includes monitoring the lock mechanism's running status, concurrency conflict detection, and data consistency validation. Transaction isolation levels include READ COMMITTED or REPEATABLE READ. The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

4. The Al-driven structured data migration method of claim 1, wherein, Non-standard structured data transformation includes the following operations: Instruction fine-tuning: A standardized domain-specific fine-tuning dataset is constructed based on the domain-specific fine-tuning dataset. Based on the standardized domain-specific fine-tuning dataset, task instruction templates, domain-specific vocabulary mapping tables, and instruction parameter configurations are extracted. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction includes unstructured data input, domain field extraction requirements, and expected output. After fine-tuning and validating the preliminary domain-adapted large language model, a domain-adapted large language model is generated. Mind Chain Reasoning: Input the unstructured data to be processed and the mind chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction. Output a preliminary set of regularized fields and mind chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files. Step-by-step analysis: Split the unstructured data to be processed according to the data segmentation rules, perform mind chain reasoning to extract rule-based fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule-based fields; Hybrid verification system filtering: Based on confidence threshold, field verification rule base and manual review triggering conditions, the precise rule-based field set is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final valid rule-based field set; Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final valid rule-based field set is converted into standard structured data supported by the target database. If subsequent steps discover field quality, completeness or compatibility issues, the feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met. Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

5. The Al-driven structured data migration method of claim 1, wherein, Data migration execution includes the following operations: Migration script generation: Based on the transformed standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization; Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules, and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified. Subtask dynamic scheduling: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted. Data splitting and parallel processing: Standardized structured data is split into shards according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data shards with source hash values; Data fragment transmission: Based on the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database synchronously or asynchronously through an encrypted transmission channel, and the transmission status is recorded; Data sharding loading: Based on the target database table structure information and data loading rules, the data transmitted to the target node is sharded and loaded into the target database in batches, supporting transaction isolation and failure retry; Data integrity verification: Based on the hash algorithm configuration, calculate the hash value of the data shards at the source and destination ends respectively, and verify data integrity by comparing the shard level and the full summary level, and handle inconsistent data; Specifically, for each execution step in the migration script, compatibility score, transformation cost, and business continuity indicators are defined as states, and each execution step is defined as an action. A Markov decision process is constructed, and the decision strategy is iteratively updated through the Q-learning algorithm to generate a global migration path planning scheme. Furthermore, the data feature alignment between the source database and the target database is achieved through a domain adaptation algorithm. The data features include data structure, data distribution, and syntax rules. The migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data fragment transmission, data fragment loading, and data integrity verification processes generate corresponding log files, namely the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data fragment transmission log, and data integrity verification log.

6. An AI-driven based structured data migration system, characterized in that, It includes a private large language model configuration module, a non-standard structured data conversion module, an unstructured data conversion module, a data migration execution module, and an exception handling and optimization module; The private large language model configuration module is used to perform the following: deploy the open-source large language model LLM and vector database locally, configure the data dialect type, the amount of data to be converted and the exception handling strategy, import the domain-specific vocabulary mapping table to build a data conversion knowledge base, and form a private large language model with domain adaptation capabilities. The non-standard structured data conversion module performs the following: For non-standard structured data in the source database, it performs lexical and syntactic analysis based on the AST parsing method, generates an abstract syntax tree (AST), traverses and optimizes the AST nodes, converts it into standard structured data supported by the target database, and outputs standard structured data after semantic verification and transaction isolation testing. The unstructured data transformation module is used to perform the following: call the large language model, extract regularized fields from the unstructured data through instruction fine-tuning, thought chain reasoning and step-by-step parsing, and after filtering by the hybrid validation system, convert the extracted regularized fields into structured data supported by the target database; The data migration execution module is used to perform the following: For the generated structured data, a large language model generates a migration script, breaks down the migration task and dynamically schedules subtasks, uses data splitting and parallel processing technology, loads the converted structured data into the target database in chunks through synchronous or asynchronous transmission, and ensures data integrity based on hash verification. For the split migration task, the migration task is taken as the action, and compatibility score, transformation cost and business continuity are taken as the states. The migration path is planned through Erkov decision, and the data feature alignment between the source database and the target database is achieved through domain adaptive algorithm. The data features include data structure, data distribution and syntax rules. The exception handling and optimization module is used to perform the following: real-time monitoring of the migration process, recording of abnormal events, automatic retrying or large model-driven repair, continuing the migration task from the point of failure, and optimizing the large language model configuration strategy and data transformation rules based on feedback data.

7. The AI-driven structured data migration system of claim 6, wherein, The private large language model configuration module is used to perform the following operations: Local deployment: Deploy the open-source large language model and vector database locally and establish a connection; Core parameter configuration: Extract dialect differences between the source and target databases to generate a data dialect configuration table. Divide the data scale levels based on the statistical information of the data to be migrated, configure the corresponding data processing strategies, define the anomaly types and handling rules during the data migration process, and form an anomaly handling rule set. Integrate the data dialect configuration table, data processing strategies, and anomaly handling rule set into an LLM parameter configuration file. The data scale levels are divided according to the number of tables and AST nodes. The processing strategies corresponding to different levels include real-time synchronization, batch synchronization, and incremental synchronization in batches. The synchronization method supports single-threaded or cluster parallel processing. The anomaly types include data format errors, transmission interruptions, verification failures, and syntax incompatibility. The corresponding handling rules include LLM automatic correction, breakpoint resumption, automatic retransmission, and manual alarms. The anomaly log recording dimensions and monitoring alarm thresholds are also configured. Knowledge base construction: Obtain domain-specific vocabulary mapping tables, historical data migration cases, and industry data conversion standards. After structuring, convert them into high-dimensional vectorized knowledge through an LLM embedding layer. Import the vectorized knowledge into a vector database and build a semantic index to construct a data conversion knowledge base that supports incremental updates. The semantic index is built based on the cosine similarity algorithm. The data conversion knowledge base is configured with an incremental update interface to support the import and index updates of newly added domain knowledge. Domain-Adaptive LLM Generation: The LLM parameter configuration file is loaded into the locally deployed open-source large language model. Real-time linkage between the large language model and the data transformation knowledge base is established through a retrieval interface. Lightweight instructions are fine-tuned based on the data transformation knowledge base. After multi-scenario testing and iterative optimization, a private large language model with domain adaptation capabilities is generated. The lightweight instruction fine-tuning is based on historical migration cases in the data transformation knowledge base to optimize the model's ability to apply domain knowledge. Multi-scenario testing includes test cases with different database combinations, different data scales, and abnormal scenarios. When processing data migration tasks, relevant domain knowledge is first obtained through the retrieval interface, and then a migration plan is generated in combination with the core parameter configuration. After the migration task is completed, new effective rules and cases are added to the data transformation knowledge base to achieve closed-loop iteration of knowledge.

8. The AI-driven structured data migration system of claim 6, wherein, The non-standard structured data conversion module is used to perform the following operations: Lexical analysis: Obtain non-standard structured data, basic syntax rules, and non-standard format adaptation rules from the source database; construct a lexical analysis dictionary; extract and standardize lexical unit tokens; and generate a token sequence. The lexical unit token includes keywords, identifiers, data types, operators, delimiters, and constants. Standardization processing includes unifying keyword case, replacing custom symbols with standard symbols, and filtering invalid characters. The output token sequence records the type, position, original value, and standardized value of each token. Syntax Analysis: Based on the Token sequence, source database syntax rules, and target database syntax compatibility rules, a hierarchical abstract syntax tree (AST) is constructed using a syntax parsing algorithm, and the integrity of the AST structure is verified. The syntax parsing algorithm includes a recursive descent algorithm and an LR parser, which are dynamically selected according to the complexity of the data structure. The node hierarchy of the AST includes a root node, first-level child nodes, and leaf nodes. The root node corresponds to the data operation type, which includes table structure definition and stored procedures. The first-level child nodes correspond to the core elements, which include field lists and index definitions. The leaf nodes correspond to specific attributes, which include field names, data types, and constraints. AST Node Traversal and Optimization: The visitor pattern is used to traverse the nodes at each level of the AST. Combining the target database syntax specifications, data type mapping rules, and constraint adaptation rules, the nodes are converted, structure optimized, constraint adapted, and redundancy is handled to generate an optimized AST. The structure optimization includes converting the source database data type nodes into target database compatible type nodes according to the mapping rules, adjusting the AST structure to adapt to the target database syntax specifications, replacing non-standard constraint nodes with constraint forms supported by the target database, deleting redundant nodes, and supplementing missing required nodes. Standard structured data generation: Based on the optimized AST, target database code generation rules and output template, standard structured data supported by the target database is generated by splicing. The standard structured data includes one or more of the following: table structure definition statements, stored procedure code, and field definition list compatible with the target database. The data is formatted and standardized during the generation process. Semantic validation and transaction isolation testing: Load the target database's semantic rules, business scenario transaction rules, and test cases. Perform semantic validity and structural integrity validation on the standard structured data. Simulate business concurrency scenarios to execute transaction isolation tests. After successful validation, output the final standard structured data. If the semantic validation or transaction isolation test fails, feedback is sent back to the corresponding preceding steps for adjustment until the test passes. Semantic validation includes field data type matching with business semantics, constraint validity validation, and logical consistency validation of associated fields. Transaction isolation testing includes monitoring the lock mechanism's running status, concurrency conflict detection, and data consistency validation. Transaction isolation levels include READ COMMITTED or REPEATABLE READ. The steps of lexical analysis, syntax analysis, AST node traversal and optimization, standard structured data generation, and semantic verification and transaction isolation testing all generate corresponding log files, namely lexical analysis log, AST structure verification report, node optimization log, code generation log, semantic verification report, and transaction isolation test report, which are used for tracing the source of the conversion process and locating anomalies.

9. The AI-driven structured data migration system according to claim 6, characterized in that, The non-standard structured data conversion module is used to perform the following operations: Instruction fine-tuning: A standardized domain-specific fine-tuning dataset is constructed based on the domain-specific fine-tuning dataset. Based on the standardized domain-specific fine-tuning dataset, task instruction templates, domain-specific vocabulary mapping tables, and instruction parameter configurations are extracted. Natural language fields in the task instruction templates are replaced with domain standard fields. Samples are traversed to generate batch standardized fine-tuning instruction sets. Each instruction includes unstructured data input, domain field extraction requirements, and expected output. After fine-tuning and validating the preliminary domain-adapted large language model, a domain-adapted large language model is generated. Mind Chain Reasoning: Input the unstructured data to be processed and the mind chain guidance instructions into the domain-adaptive large language model, and decompose the reasoning according to the operation logic of core information positioning, field matching, format verification, and logical correction. Output a preliminary set of regularized fields and mind chain reasoning log. The unstructured data to be processed includes text, XML documents, images, emails and log files. Step-by-step analysis: Split the unstructured data to be processed according to the data segmentation rules, perform mind chain reasoning to extract rule-based fields for each sub-data block, merge and integrate the extraction results of all sub-data blocks, and generate a precise set of rule-based fields; Hybrid verification system filtering: Based on confidence threshold, field verification rule base and manual review triggering conditions, the precise rule-based field set is subjected to confidence verification, rule verification and manual review to filter invalid fields and output the final valid rule-based field set; Structured data conversion: Based on the target database configuration information, structured data output template and data type mapping rules, the final valid rule-based field set is converted into standard structured data supported by the target database. If subsequent steps discover field quality, completeness or compatibility issues, the feedback is sent back to the corresponding preceding steps for iterative optimization until the target requirements are met. Both the thought chain reasoning and the step-by-step parsing generate corresponding log files, namely the thought chain reasoning log and the step-by-step parsing log, which are used to record the reasoning process, data block details, extraction basis, and integration records.

10. The AI-driven structured data migration system according to claim 6, characterized in that, The data migration execution module is used to perform the following operations: Migration script generation: Based on the transformed standardized structured data, the configuration information of the source database and the target database, and the migration requirement parameters, a set of migration scripts compatible with the target database is generated through a domain-adaptive large language model, and output after syntax verification and efficiency optimization; Migration task breakdown: Based on the migration script set, structured data statistics, task breakdown rules, and target database cluster node information, the overall migration task is broken down into a set of independent subtasks with dependencies, and the priority and execution order of the subtasks are clarified. Subtask dynamic scheduling: Based on the subtask set, the real-time status of the target database cluster resources and the scheduling strategy configuration, execution resources are dynamically allocated, the execution status of subtasks is monitored in real time, exceptions are handled according to rules and the scheduling scheme is dynamically adjusted. Data splitting and parallel processing: Standardized structured data is split into shards according to data splitting rules, and data preprocessing and compression are performed in parallel by multiple threads / multi-nodes to generate structured data shards with source hash values; Data fragment transmission: Based on the transmission mode configuration, data fragments are transmitted to the corresponding nodes of the target database synchronously or asynchronously through an encrypted transmission channel, and the transmission status is recorded; Data sharding loading: Based on the target database table structure information and data loading rules, the data transmitted to the target node is sharded and loaded into the target database in batches, supporting transaction isolation and failure retry; Data integrity verification: Based on the hash algorithm configuration, calculate the hash value of the data shards at the source and destination ends respectively, and verify data integrity by comparing the shard level and the full summary level, and handle inconsistent data; Among them, the migration script generation, migration task decomposition, subtask dynamic scheduling, data splitting and parallel processing, data shard transmission, data shard loading, and data integrity verification are executed, and corresponding log files are generated. These are the migration script generation log, migration task decomposition log, subtask dynamic scheduling log, data splitting and parallel processing log, data shard transmission log, and data integrity verification log.

Images