Data collection and mapping method, system, device and medium of multi-type database

Abstract

The application relates to the technical field of data flow conversion, in particular to a data collection and mapping method, system, device and medium for multiple types of databases, which comprises the following steps: collecting to-be-mapped data from a source database; using a large language model to analyze the field name of the to-be-mapped data, deducing the field type, and pre-processing the to-be-mapped data; using a reinforcement learning optimization algorithm to generate a data matching rule between the source database and a target database; converting the fields of the pre-processed to-be-mapped data according to the data matching rule to obtain mapping data, and storing the mapping data into the target database. The application can reduce manual intervention, improve matching accuracy, reduce error correction lag and manual troubleshooting cost, realize efficient parallelization of tasks, and enhance system maintainability and expansion.

Description

Technical Field

[0001] This application relates to the field of data flow technology, specifically to data acquisition and mapping methods, systems, devices and media for various types of databases. Background Technology

[0002] With the acceleration of enterprise digital transformation and the explosive growth of data volume, database types are becoming increasingly diversified, encompassing various forms such as relational databases, non-relational databases, and time-series databases. The need for cross-platform and cross-structure data acquisition and mapping is becoming increasingly urgent, especially in data migration, system integration, and data analysis scenarios. Efficiently and accurately realizing data flow between heterogeneous databases has become a core challenge for enterprises.

[0003] In existing technologies, traditional methods mainly achieve data collection and field mapping through manual rule configuration or static scripts. For example, data migration is completed through predefined data transformation rule tables or fixed ETL (Extract-Transform-Load) processes. These methods rely on human experience, use scripts to match the field types and formats of the source and target databases, and introduce rule engines to assist in data cleaning and standardization.

[0004] However, existing data acquisition and field mapping methods rely heavily on manual intervention. When dealing with dynamically changing database structures or new data types, rules and scripts need to be frequently adjusted, resulting in high maintenance costs and insufficient flexibility. At the same time, traditional rule configuration methods are difficult to cover the heterogeneous characteristics of multiple database types and lack real-time error detection and self-correction capabilities, leading to a high risk of error accumulation during data migration. Furthermore, static scripts and fixed rules are difficult to optimize themselves based on historical experience, and cannot achieve intelligent task scheduling and resource allocation, resulting in significant bottlenecks in overall efficiency and accuracy. Summary of the Invention

[0005] To address the technical challenges of existing data acquisition and mapping methods that heavily rely on manual intervention, resulting in high maintenance costs, insufficient flexibility, lack of real-time error detection and self-correction capabilities, difficulty in self-optimization, and non-intelligent task scheduling and resource allocation when facing dynamic changes in database structure, new data types, and heterogeneous features of multiple database types, thus limiting overall efficiency and accuracy, this application provides a data acquisition and mapping method, system, device, and medium for multiple database types. This method can reduce manual intervention, improve matching accuracy, reduce error correction delays and manual troubleshooting costs, achieve efficient task parallelization, and enhance system maintainability and scalability.

[0006] Firstly, this application provides a method for data acquisition and mapping of multiple types of databases, comprising the following steps: S1. Collect the data to be mapped from the source database. The source database types include relational databases, non-relational databases, and time-series databases. S2. Use a large language model to analyze the field names of the data to be mapped and infer the field types; The data to be mapped is preprocessed, including data cleaning and standardization. S3. Use reinforcement learning optimization algorithms to generate data matching rules between the source database and the target database. The data matching rules include matching rules for field types and field lengths. S4. Based on the data matching rules, transform the fields of the preprocessed data to be mapped to obtain the mapped data, and store the mapped data in the target database.

[0007] It should be further noted that in step S1, the source database type includes MySQL, PostgreSQL, MongoDB, Redis or InfluxDB, and the connection to the source database is achieved through JDBC, ODBC, REST API or GraphQL interface. Supported data formats include JSON, CSV or XML.

[0008] It should be further explained that step S1 achieves data collection through a scheduling mechanism, which includes a timed collection strategy based on Cron expressions, a task queue combined with event-driven methods, adjusting the collection frequency through dynamic load balancing, and multi-threaded parallel collection.

[0009] It should be further noted that step S1 uses Kafka or RabbitMQ to implement an event-driven task queue.

[0010] It should be further noted that in step S2, the large language model is a BERT or GPT model in which the original output layer is replaced with a fully connected classification layer, and the output dimension of the large language model is equal to the number of categories of the field type.

[0011] It should be further noted that the preprocessing steps in step S2 include: The contrastive learning method in self-supervised learning is used to extract field features from the collected data, and the K-means clustering algorithm is combined to automatically adjust the category division according to the field distribution of the data to be mapped. Data cleaning rules are executed through the Drools rule engine. Specific rules include detecting and removing duplicate values, filling in missing values, standardizing date formats, and unifying numerical formats.

[0012] It should be further noted that in step S3, the reinforcement learning optimization algorithm employs either a Deep Q-Network (DQN) or a Proximal Policy Optimization (PPO) algorithm, and generates data matching rules based on the following mechanism: Construct a reinforcement learning agent, whose state space is defined by the field names, types, lengths, and data distributions of the source and target databases; Define the action space, including field matching, adjusting field types, and applying transformation rules; The reward mechanism is dynamically adjusted based on the matching accuracy, with a positive reward for a successful match and a negative reward for a failed match. The agent stores historical data migration logs through an Experience Replay buffer and generalizes the experience to new database migration tasks by combining meta-learning methods.

[0013] It should be further explained that in step S4, the preprocessed data fields to be mapped are transformed according to the data matching rules, which specifically includes the following steps: S401. Based on the Abstract Syntax Tree (AST) tool, parse the SQL syntax structure of the target database and generate conversion logic that is compatible with the field type and length requirements in the data matching rules; S402. By using the few-shot learning method and the field type mapping relationship in the data matching rules, the conversion requirements of new field types can be adapted. S403. Generate a data mapping script using the Apache Beam framework, and dynamically populate the data matching rules into an SQL (Structured Query Language) or ETL (Extract-Transform-Load) script using the Jinja2 template engine. Transform the preprocessed fields of the data to be mapped to obtain the mapped data.

[0014] It should be further noted that during the execution of steps S1-S4, errors are detected in real time and automatically corrected.

[0015] It should be further noted that the real-time error detection and automatic correction operations include: The fine-tuned BERT model is used to detect field type mismatches or data format errors, and the error logs are analyzed using the Isolation Forest algorithm, combined with reinforcement learning to dynamically correct the mapping rules.

[0016] It should be further noted that after each execution of steps S1-S4, the following optimization steps are performed: Based on the field type classification results output by the large language model and the actual field types of the target database, the classification accuracy is calculated, and combined with the field value accuracy of the data transformation, the parameters of the large language model are updated through the backpropagation algorithm. Based on task execution efficiency, resource consumption, and the load status of the target database, dynamically adjust the priority of data collection and resource allocation strategy; The updated large language model parameters and adjusted resource allocation strategies will be applied to the next data acquisition and mapping task.

[0017] It should be further explained that updating the parameters of a large language model through federated learning specifically includes: A joint loss function is constructed based on classification accuracy and field value accuracy, and the gradient of the large language model is calculated locally for each data migration instance. The model gradients of each instance are aggregated using a weighted average algorithm, with the weights determined by the amount of data and historical classification accuracy of each instance. Before the gradient is uploaded to the central server, Gaussian noise that satisfies (ε, δ)-differential privacy is added; The multi-armed bandit algorithm dynamically adjusts data acquisition priorities and resource allocation strategies, specifically including: Each data collection task is defined as an independent "arm," and the expected reward for each task is dynamically calculated based on the classification accuracy.

[0018] Secondly, the present invention provides a data acquisition and mapping system for multiple types of databases, used to implement the above-mentioned data acquisition and mapping method for multiple types of databases, including: The data acquisition module is used to collect the data to be mapped from the source database; The field type inference module is used to analyze the field names of the data to be mapped using a large language model, infer the field types, and preprocess the data to be mapped. The matching rule generation module is used to generate data matching rules between the source database and the target database using reinforcement learning optimization algorithms. The data conversion and storage module converts the fields of the preprocessed data to be mapped according to the data matching rules to obtain the mapped data, and stores the mapped data in the target database.

[0019] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned data acquisition and mapping method for multiple types of databases.

[0020] Fourthly, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned data acquisition and mapping method for multiple types of databases.

[0021] As can be seen from the above technical solutions, this application has the following advantages: 1. This application analyzes the field names and data features of the source and target databases using a large language model, and dynamically optimizes the matching strategy using reinforcement learning algorithms to automatically generate field type mapping rules. Compared with traditional methods of manually configuring rules, this significantly reduces human intervention, improves matching accuracy, adapts to diverse field naming rules, effectively handles complex or non-standardized data naming scenarios, and ensures the accuracy and consistency of field type identification. At the same time, the automated data cleaning and standardization process eliminates subjective biases in manual operations, improves the quality and efficiency of data preprocessing, and fundamentally reduces the risk of errors caused by non-standard data during data migration.

[0022] 2. This application utilizes reinforcement learning optimization algorithms to dynamically generate matching rules based on the field features of the source and target databases, overcoming the rigidity of traditional static scripts or fixed rules when faced with changes in database structure. At the same time, it autonomously optimizes the matching strategy by continuously learning feedback information from historical migration tasks, achieving rapid adaptation to new field types and heterogeneous data structures, and significantly improving the robustness of the system in dynamic data environments.

[0023] 3. This application automates the entire process from data collection and preprocessing to matching rule generation and conversion, reducing manual intervention and avoiding the transmission and accumulation of errors caused by operational delays or human oversights in traditional segmented processing. Through real-time data cleaning, standardization, and error detection mechanisms, the system can identify and correct abnormal data in the early stages, ensuring the standardization and integrity of the data before mapping, thereby reducing the error rate in the subsequent conversion stages.

[0024] 4. This application supports the access and conversion of mixed data sources including relational, non-relational, and time-series databases, breaking through the traditional method's dependence on a single database type. Through modular design and universal interface protocols, the system can be flexibly extended to new database types to meet the diverse data integration needs of enterprises and improve the long-term applicability of the technical solution.

[0025] 5. This application extracts data patterns from historical tasks through self-supervised learning and combines this with large-scale model anomaly detection technology to identify data format errors, field type mismatches, and other issues in real time, automatically correcting them or generating feedback logs. This mechanism solves the problem of lagging error correction in traditional methods, reduces the cost of manual troubleshooting, and continuously optimizes the model through incremental learning, improving its robustness to subsequent tasks. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a data acquisition and mapping method for multiple types of databases in one embodiment of the present invention.

[0028] Figure 2 This is a schematic block diagram of a data acquisition and mapping system for multiple types of databases in one embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of the present invention. Detailed Implementation

[0030] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this patent.

[0031] This application's data acquisition and mapping method for multiple database types primarily targets the field of data transfer technology. It analyzes the field names and data characteristics of the source and target databases using a large language model, and dynamically optimizes the matching strategy using reinforcement learning algorithms to automatically generate field type mapping rules. Compared to traditional methods of manually configuring rules, this significantly reduces human intervention, improves matching accuracy, adapts to diverse field naming rules, effectively handles complex or non-standardized data naming scenarios, and ensures the accuracy and consistency of field type identification. Simultaneously, the automated data cleaning and standardization process eliminates subjective biases in manual operations, improves the quality and efficiency of data preprocessing, and fundamentally reduces the risk of errors caused by non-standard data during data migration. Utilizing reinforcement learning optimization algorithms, it dynamically generates matching rules based on the field characteristics of the source and target databases, overcoming the rigidity of traditional static scripts or fixed rules when facing changes in database structure. By continuously learning from feedback from historical migration tasks, the system autonomously optimizes its matching strategy, enabling rapid adaptation to new field types and heterogeneous data structures, significantly improving its robustness in dynamic data environments. The fully automated process, from data acquisition and preprocessing to matching rule generation and conversion, reduces manual intervention and avoids the transmission and accumulation of errors caused by operational delays or human oversights in traditional segmented processing. Through real-time data cleaning, standardization, and error detection mechanisms, the system can identify and correct abnormal data in the early stages, ensuring the standardization and integrity of data before mapping, thereby reducing the error rate in subsequent conversion stages. It supports mixed data source access and conversion for relational, non-relational, and time-series databases, breaking through the traditional limitations of relying on a single database type. Through modular design and universal interface protocols, the system can be flexibly extended to new database types, meeting diverse data integration needs of enterprises and enhancing the long-term applicability of the technical solution.

[0032] The data acquisition and mapping method for multiple types of databases involved in this application mainly addresses the technical problems of existing data acquisition and mapping methods that rely heavily on manual intervention. When faced with dynamic changes in database structure, new data types, and heterogeneous characteristics of multiple types of databases, these methods suffer from high maintenance costs, insufficient flexibility, lack of real-time error detection and self-correction capabilities, difficulty in self-optimization, and non-intelligent task scheduling and resource allocation, which limit the overall efficiency and accuracy.

[0033] The following describes in detail the data acquisition and mapping methods for various types of databases involved in this application. Specific details, such as particular system structures and technologies, are presented for illustrative purposes rather than limiting, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0034] In the data acquisition and mapping method for multiple types of databases involved in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or sets thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0035] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0036] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The data acquisition and mapping method for multiple types of databases provided in this embodiment of the invention is executed by a computer device, and correspondingly, the data acquisition and mapping system for multiple types of databases runs on the computer device.

[0039] Figure 1 This is a flowchart of a data acquisition and mapping method for multi-type databases according to an embodiment of the present invention. Figure 1 The executing entity can be a data acquisition and mapping system for multiple types of databases. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.

[0040] like Figure 1 As shown, the data acquisition and mapping method for this multi-type database includes: Step S1: Collect the data to be mapped from the source database. The source database types include relational databases, non-relational databases, and time-series databases.

[0041] By supporting the collection of relational databases, non-relational databases, and time-series databases, it overcomes the limitations of traditional methods that only target a single database type. The unified collection process reduces the development and maintenance costs of adapting to multiple database types, improves the universality of cross-platform data migration, and ensures that enterprises can efficiently integrate heterogeneous data sources to meet the diverse needs of complex business scenarios.

[0042] In some specific embodiments, the source database type includes MySQL, PostgreSQL, MongoDB, Redis, or InfluxDB, and the connection to the source database is implemented through JDBC, ODBC, REST API, or GraphQL interface. Supported data formats include JSON, CSV, or XML.

[0043] By limiting the database type (such as MySQL, MongoDB) and connection interface (JDBC, REST API, etc.), the technical solution ensures broad compatibility with mainstream databases, supports flexible exchange of common data formats such as JSON, CSV, and XML, meets the access needs of heterogeneous data sources, reduces the technical threshold for enterprise multi-system integration, and improves the standardization and scalability of data migration.

[0044] In some specific embodiments, data acquisition is achieved through a scheduling mechanism, which includes a timed acquisition strategy based on Cron expressions, a task queue combined with event-driven approaches, dynamic load balancing to adjust the acquisition frequency, and multi-threaded parallel acquisition.

[0045] The scheduling mechanism combines timed data collection, event-driven task queues, dynamic load balancing, and multi-threaded parallel data collection to optimize resource allocation and task execution efficiency. The event-driven mechanism supports real-time data triggering, dynamic load balancing handles high-concurrency scenarios, and multi-threaded parallel processing shortens the data collection cycle, thereby improving the throughput and response speed of large-scale data migration.

[0046] In some specific implementations, Kafka or RabbitMQ is used to implement a task queue that combines event-driven operations.

[0047] Implement event-driven task queues using Kafka or RabbitMQ, leveraging their high reliability, distributed architecture, and fault tolerance to ensure the stability and scalability of the data collection process, avoid data loss or duplicate processing, and support horizontal scaling to adapt to the real-time synchronization needs of massive amounts of data.

[0048] Step S2: Use a large language model to analyze the field names of the data to be mapped and infer the field types; The data to be mapped is preprocessed, including data cleaning and standardization.

[0049] By using a large language model to automatically analyze the semantic information of field names and infer field types, the traditional rule configuration driven by human experience is replaced, which significantly improves the accuracy and efficiency of field type identification. Through data cleaning and standardization, noise in the original data (such as missing values ​​and disordered format) is eliminated, ensuring data standardization from the source and reducing the risk of errors caused by data quality issues in the subsequent mapping process.

[0050] In some specific implementations, the large language model is a BERT or GPT model in which the original output layer is replaced with a fully connected classification layer, and the output dimension of the large language model is equal to the number of categories of the field type.

[0051] By fine-tuning the architecture of BERT or GPT models, high-precision semantic inference of field types can be achieved. The models can quickly adapt to non-standardized field naming rules in different fields, reduce the need for manual intervention, improve the accuracy and consistency of field type identification, and lay a reliable foundation for subsequent data processing.

[0052] In some specific embodiments, the preprocessing steps include: The contrastive learning method in self-supervised learning is used to extract field features from the collected data, and the K-means clustering algorithm is combined to automatically adjust the category division according to the field distribution of the data to be mapped. Data cleaning rules are executed through the Drools rule engine. Specific rules include detecting and removing duplicate values, filling in missing values, standardizing date formats, and unifying numerical formats.

[0053] The preprocessing process combines self-supervised learning to extract field features, uses K-means clustering to automatically adjust the data category division, and executes cleaning rules through the Drools rule engine to automate data cleaning and standardization. Self-supervised learning mines potential patterns from unlabeled data, and the rule engine supports flexible configuration of cleaning logic to ensure data standardization and reduce manual cleaning costs and subjective biases.

[0054] Step S3: Use reinforcement learning optimization algorithms to generate data matching rules between the source database and the target database. The data matching rules include matching rules for field types and field lengths.

[0055] The system uses reinforcement learning optimization algorithms to dynamically generate data matching rules, overcoming the shortcomings of traditional static rules or scripts that cannot adapt to changes in database structure. Through real-time feedback and historical task experience, the system can autonomously adjust the matching strategy for field types and lengths, improving the flexibility and adaptability of rule generation and ensuring high-precision mapping even when faced with new fields or changes in data structure.

[0056] In some specific embodiments, the reinforcement learning optimization algorithm employs a deep Q-network (DQN) or a proximal policy optimization (PPO) algorithm, and generates data matching rules based on the following mechanism: Construct a reinforcement learning agent, whose state space is defined by the field names, types, lengths, and data distributions of the source and target databases; Define the action space, including field matching, adjusting field types, and applying transformation rules; The reward mechanism is dynamically adjusted based on the matching accuracy, with a positive reward for a successful match and a negative reward for a failed match. The agent stores historical data migration logs through an Experience Replay buffer and generalizes the experience to new database migration tasks by combining meta-learning methods.

[0057] Reinforcement learning algorithms employ DQN or PPO, defining state and action spaces. They dynamically optimize matching strategies using reward mechanisms and generalize historical experience to new tasks through meta-learning, significantly improving field matching accuracy and adaptability across database scenarios. A replay buffer stores migration logs, supporting continuous agent learning, reducing repetitive training costs, and enhancing long-term system stability.

[0058] Step S4: Based on the data matching rules, the fields of the preprocessed data to be mapped are transformed to obtain the mapped data, and the mapped data is stored in the target database.

[0059] This approach automatically converts fields based on data matching rules and stores them in the target database, solving the problems of low efficiency and error-proneness caused by manually writing conversion scripts in traditional methods. The automated conversion process reduces human intervention, ensuring the accuracy and consistency of the mapping results. At the same time, it achieves seamless integration with the target database through standardized processes, significantly improving the overall efficiency and reliability of data migration.

[0060] In some specific embodiments, the specific steps for the conversion are as follows: S401. Based on the Abstract Syntax Tree (AST) tool, parse the SQL syntax structure of the target database and generate conversion logic that is compatible with the field type and length requirements in the data matching rules; S402. By using the few-shot learning method and the field type mapping relationship in the data matching rules, the conversion requirements of new field types can be adapted. S403. Generate a data mapping script using the Apache Beam framework, and dynamically populate the data matching rules into an SQL (Structured Query Language) or ETL (Extract-Transform-Load) script using the Jinja2 template engine. Transform the preprocessed fields of the data to be mapped to obtain the mapped data.

[0061] Field type conversion is based on AST tools to parse the SQL syntax structure, ensuring the syntactic correctness of the generated script. It combines small sample learning to quickly adapt to new field type requirements. Standardized scripts are automatically generated through Apache Beam and Jinja2 template engines, reducing manual coding errors and improving conversion efficiency and maintainability.

[0062] In some specific embodiments, during the execution of steps S1-S4, errors are detected in real time and automatically corrected.

[0063] The end-to-end real-time error detection and automatic correction mechanism instantly identifies anomalies (such as type mismatch or format error) during data acquisition, matching, and conversion, interrupting the error propagation chain. Combined with a closed-loop correction strategy, it dynamically adjusts rules, significantly reducing the risk of error accumulation and improving the overall reliability of data migration.

[0064] In some specific implementations, a fine-tuned BERT model is used to detect field type mismatches or data format errors, and error logs are analyzed using the Isolation Forest algorithm, combined with reinforcement learning to dynamically correct mapping rules.

[0065] By fine-tuning the BERT model to detect semantic errors and combining it with the Isolation Forest algorithm to analyze statistical anomalies, high-precision error localization is achieved. Reinforcement learning is used to dynamically optimize mapping rules through a linkage correction strategy, thereby reducing the incidence of similar errors in the long term, forming a self-optimizing error handling closed loop, and improving the robustness of the system.

[0066] In some specific embodiments, after each execution of steps S1-S4, the following optimization steps are performed: Based on the field type classification results output by the large language model and the actual field types of the target database, the classification accuracy is calculated, and combined with the field value accuracy of the data transformation, the parameters of the large language model are updated through the backpropagation algorithm. Based on task execution efficiency, resource consumption, and the load status of the target database, dynamically adjust the priority of data collection and resource allocation strategy; The updated large language model parameters and adjusted resource allocation strategies will be applied to the next data acquisition and mapping task.

[0067] After each task is executed, the model parameters and resource allocation strategy are updated. The model performance is continuously optimized through incremental learning, and the data collection priority and computing resource allocation are dynamically adjusted to ensure that the system gradually improves efficiency and accuracy as tasks accumulate, so as to adapt to ever-changing business needs.

[0068] In some specific embodiments, parameter updates for large language models are achieved through federated learning, specifically including: A joint loss function is constructed based on classification accuracy and field value accuracy, and the gradient of the large language model is calculated locally for each data migration instance. The model gradients of each instance are aggregated using a weighted average algorithm, with the weights determined by the amount of data and historical classification accuracy of each instance. Before the gradient is uploaded to the central server, Gaussian noise that satisfies (ε, δ)-differential privacy is added; The multi-armed bandit algorithm dynamically adjusts data acquisition priorities and resource allocation strategies, specifically including: Each data collection task is defined as an independent "arm," and the expected reward for each task is dynamically calculated based on the classification accuracy.

[0069] Federated learning integrates multi-instance experience to optimize model parameters, improves model generalization ability and cross-scenario applicability, and uses multi-armed slot machine algorithms to dynamically schedule resources, balance exploration and utilization, prioritize high-value tasks, maximize global resource utilization, and significantly improve the overall system throughput and task success rate.

[0070] In one specific embodiment, the data acquisition and mapping method for multiple types of databases includes: Step S1: Collect the data to be mapped from the source database. The source database types include relational databases, non-relational databases, and time-series databases. The source database can be of the type MySQL, PostgreSQL, MongoDB, Redis or InfluxDB. The connection to the source database is achieved through JDBC, ODBC, REST API or GraphQL interface. The supported data formats include JSON, CSV or XML. Data collection is achieved through a scheduling mechanism, which includes a timed collection strategy based on Cron expressions, a task queue that combines event-driven processing using Kafka or RabbitMQ, dynamic load balancing to adjust the collection frequency, and multi-threaded parallel collection. Step S2: Use a large language model to analyze the field names of the data to be mapped and infer the field types; The data to be mapped is preprocessed, including data cleaning and standardization. The architecture of the large language model is as follows: The BERT model is used as the basic architecture, which adopts the Transformer architecture with 12 Transformer blocks and a hidden layer dimension of 768. For the field type inference task, the BERT output layer is removed and replaced with a fully connected classification layer. The output dimension of the large language model is equal to the number of field types. The output categories include common field types, including integers, floating-point numbers, dates, text, and booleans. The training steps for a large language model include: S201. Training data preparation: Extract field names and their corresponding type labels from public database schema libraries (such as Open Database Schema), covering multiple fields such as finance, healthcare, and e-commerce, and construct the training set and validation set in an 8:2 ratio; Expand the training set in the following ways: Field name variant generation: For example, expanding "user_id" to "userID", "UserID", "uid", etc.

[0071] Contextual population: Add contextual descriptions to field names, such as expanding "price" to "commodity price (unit: yuan)"; The input for each sample is the field name and its context information, and the output is the field type label; S202. Model fine-tuning using the AdamW optimizer, with an initial learning rate set to 2e. -5 The batch size is 32, the training cycle is 5 rounds, and a linear learning rate decay strategy is used. Contrastive learning is employed on unlabeled data, using the SimCSE framework to generate positive and negative sample pairs of field names, thereby enhancing the model's ability to capture semantic similarity. The goal of model fine-tuning is to achieve an accuracy of 95.3% on the validation set for the large language model; Preprocessing steps include: The contrastive learning method in self-supervised learning is used to extract field features from the collected data, and the K-means clustering algorithm is combined to automatically adjust the category division according to the field distribution of the data to be mapped. Data cleaning rules are executed through the Drools rule engine. Specific rules include detecting and removing duplicate values, filling missing values, standardizing date formats, and unifying numerical formats. Step S3: Use reinforcement learning optimization algorithm to generate data matching rules between source database and target database. The data matching rules include matching rules for field type and field length. The reinforcement learning optimization algorithm employs the Deep Q-Network (DQN) algorithm and generates data matching rules based on the following mechanism: Construct a reinforcement learning agent, whose state space is defined by the field names, types, lengths, and data distributions of the source and target databases; Define the action space, including field matching, adjusting field types, and applying transformation rules; The reward mechanism is dynamically adjusted based on the matching accuracy. When a match is successful and there are no errors after data conversion, a +1 reward is given. A reward of -1 is given when a match fails or data loss / formatting error occurs after conversion; When the agent tries a new matching strategy, it is given a small positive reward of +0.2 based on the strategy entropy value; The agent stores historical data migration logs through an Experience Replay buffer and generalizes the experience to new database migration tasks by combining meta-learning methods. The agent training process for reinforcement learning optimization algorithms is as follows: S301. Set the Experience Replay buffer to store the state-action-reward triplet in the historical migration log, and set the capacity of the Experience Replay buffer to 100,000 entries. S302. The MAML (Model-Agnostic Meta Learning) algorithm is used to train the agent across multiple heterogeneous database migration tasks (such as MySQL→PostgreSQL, MongoDB→InfluxDB) to enable the agent to quickly adapt to new tasks. S303. Train the agent until convergence. In this embodiment, after 200,000 iterations, the matching accuracy is improved from the initial 70% to 92%, and the average reward is stabilized at above 0.8. Step S4: Based on the data matching rules, transform the fields of the preprocessed data to be mapped to obtain the mapped data. This specifically includes the following steps: S401. Based on the Abstract Syntax Tree (AST) tool, parse the SQL syntax structure of the target database and generate conversion logic that is compatible with the field type and length requirements in the data matching rules; S402. By using the few-shot learning method and the field type mapping relationship in the data matching rules, the conversion requirements of new field types can be adapted. S403. Generate a data mapping script using the Apache Beam framework, and dynamically populate the data matching rules into an SQL (Structured Query Language) or ETL (Extract-Transform-Load) script using the Jinja2 template engine. Transform the preprocessed fields of the data to be mapped to obtain the mapped data. Store the mapped data in the target database; During the execution of steps S1-S4, a fine-tuned BERT model is used to detect field type mismatches or data format errors, and error logs are analyzed using the Isolation Forest algorithm, combined with reinforcement learning to dynamically correct mapping rules; The fine-tuning of the BERT model includes: Task definition: Model error detection as a binary classification task (normal / abnormal); Training data: A labeled dataset is constructed using error cases (such as type mismatches and format errors) from historical migration logs; Fine-tuning parameters: Set the learning rate to 3e -5 With a batch size of 16 and training using the cross-entropy loss function, the accuracy reached 98.5% after three rounds of training. Error log analysis using the Isolation Forest algorithm includes: Input the field value distribution characteristics (such as numerical range and dispersion) during the data migration process into the isolated forest model to identify statistical anomalies; Based on the cumulative distribution of error logs, the anomaly scoring threshold is dynamically adjusted to reduce the false alarm rate; Error correction strategies include: when an error is detected, triggering the agent to regenerate matching rules and penalizing the erroneous action through a reward mechanism to optimize the long-term strategy; After each execution of steps S1-S4, perform the following optimization steps: Based on the field type classification results output by the large language model and the actual field types of the target database, the classification accuracy is calculated, and combined with the field value accuracy of the data transformation, the parameters of the large language model are updated through the backpropagation algorithm. Based on task execution efficiency, resource consumption, and the load status of the target database, dynamically adjust the priority of data collection and resource allocation strategy; The updated large language model parameters and adjusted resource allocation strategy will be applied to the next data collection and mapping task; Specifically, updating the parameters of a large language model through federated learning includes: A joint loss function is constructed based on classification accuracy and field value accuracy, and the gradient of the large language model is calculated locally for each data migration instance. The model gradients of each instance are aggregated using a weighted average algorithm, with the weights determined by the amount of data and historical classification accuracy of each instance. Before the gradient is uploaded to the central server, Gaussian noise that satisfies (ε, δ)-differential privacy is added; The framework for federated learning is designed as follows: Participants: Multiple data migration instances are used as federated nodes, with each node training a large language model and reinforcement learning algorithm locally; Parameter aggregation: The FedAvg algorithm is used to aggregate the model parameters of each node after each round of training and update the global model. Privacy protection: Gaussian noise (σ=0.5) is added during parameter upload using differential privacy technology to ensure data privacy; The multi-armed bandit algorithm dynamically adjusts data acquisition priorities and resource allocation strategies, specifically including: Task priority allocation: Each data collection task is defined as an independent "arm", and the expected reward for each task is dynamically calculated based on the classification accuracy.

[0072] Algorithm selection: The Thompson Sampling strategy is adopted to balance exploration and utilization, and to prioritize the allocation of resources to tasks with high success probability. Real-time adjustment: The reward distribution is updated after every 100 tasks are completed to ensure that the scheduling strategy adapts to dynamic load.

[0073] The following are embodiments of the data acquisition and mapping system for multiple types of databases provided in this disclosure. This active power load reduction optimization system and the data acquisition and mapping method for multiple types of databases in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the data acquisition and mapping system for multiple types of databases, please refer to the embodiments of the data acquisition and mapping method for multiple types of databases described above.

[0074] Mobile terminals implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of illustrating the embodiments of the present invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0075] like Figure 2 As shown, the data acquisition and mapping system for multiple database types includes: The data acquisition module is used to collect the data to be mapped from the source database; The field type inference module is used to analyze the field names of the data to be mapped using a large language model, infer the field types, and preprocess the data to be mapped. The matching rule generation module is used to generate data matching rules between the source database and the target database using reinforcement learning optimization algorithms. The data conversion and storage module converts the fields of the preprocessed data to be mapped according to the data matching rules to obtain the mapped data, and stores the mapped data in the target database.

[0076] The data acquisition and mapping system in this embodiment is used to implement a data acquisition and mapping method for multiple types of databases, and the steps include: S1. Collect the data to be mapped from the source database. The source database types include relational databases, non-relational databases, and time-series databases. S2. Use a large language model to analyze the field names of the data to be mapped and infer the field types; The data to be mapped is preprocessed, including data cleaning and standardization. S3. Use reinforcement learning optimization algorithms to generate data matching rules between the source database and the target database. The data matching rules include matching rules for field types and field lengths. S4. Based on the data matching rules, transform the fields of the preprocessed data to be mapped to obtain the mapped data, and store the mapped data in the target database.

[0077] This application also provides an electronic device for implementing various embodiments of the present invention, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0078] Those skilled in the art will understand that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0079] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0080] Electronic devices include, but are not limited to, components such as processors and memory. Those skilled in the art will understand that the electronic device structures described in the embodiments of this invention do not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or may combine certain components, or have different component arrangements.

[0081] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0082] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0083] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0084] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0085] This application also provides a storage medium storing a program product capable of implementing data acquisition and mapping methods for multiple types of databases. In some possible implementations, various aspects of this disclosure can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0086] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data acquisition and mapping of multiple types of databases, characterized in that, Includes the following steps: S1. Collect the data to be mapped from the source database. The source database types include relational databases, non-relational databases, and time-series databases. S2. Use a large language model to analyze the field names of the data to be mapped and infer the field types; The data to be mapped is preprocessed, including data cleaning and standardization. S3. Use reinforcement learning optimization algorithms to generate data matching rules between the source database and the target database. The data matching rules include matching rules for field types and field lengths. The reinforcement learning optimization algorithm employs deep Q-networks or proximal policy optimization algorithms, and generates data matching rules based on the following mechanism: Construct a reinforcement learning agent, whose state space is defined by the field names, types, lengths, and data distributions of the source and target databases; Define the action space, including field matching, adjusting field types, and applying transformation rules; The reward mechanism is dynamically adjusted based on the matching accuracy, with a positive reward for a successful match and a negative reward for a failed match. The agent stores historical data migration logs through a replay buffer and generalizes the experience to new database migration tasks by combining meta-learning methods. S4. Based on the data matching rules, transform the fields of the preprocessed data to be mapped to obtain the mapped data, and store the mapped data in the target database; After each execution of steps S1-S4, perform the following optimization steps: Based on the field type classification results output by the large language model and the actual field types of the target database, the classification accuracy is calculated, and combined with the field value accuracy of the data transformation, the parameters of the large language model are updated through the backpropagation algorithm. Based on task execution efficiency, resource consumption, and the load status of the target database, dynamically adjust the priority of data collection and resource allocation strategy; The updated large language model parameters and adjusted resource allocation strategy will be applied to the next data collection and mapping task; Updating parameters of a large language model through federated learning includes: A joint loss function is constructed based on classification accuracy and field value accuracy, and the gradient of the large language model is calculated locally for each data migration instance. The model gradients of each instance are aggregated using a weighted average algorithm, with the weights determined by the amount of data and historical classification accuracy of each instance. Before the gradient is uploaded to the central server, Gaussian noise that satisfies (ε, δ)-differential privacy is added; The multi-armed slot machine algorithm dynamically adjusts data collection priorities and resource allocation strategies, specifically including: Each data collection task is defined as an independent "arm", and the expected reward for each task is dynamically calculated based on the classification accuracy.

2. The data acquisition and mapping method as described in claim 1, characterized in that, In step S2, the large language model is a BERT or GPT model with the original output layer replaced by a fully connected classification layer. The output dimension of the large language model is equal to the number of categories of the field type.

3. The data acquisition and mapping method as described in claim 1, characterized in that, In step S4, the preprocessed data fields to be mapped are transformed according to the data matching rules, specifically including the following steps: S401. Based on the abstract syntax tree tool, parse the SQL syntax structure of the target database and generate conversion logic that is compatible with the field type and length requirements in the data matching rules; S402. By using a few-sample learning method, the field type mapping relationship in the data matching rules is utilized to adapt to the conversion requirements of new field types; S403. Generate a data mapping script using the Apache Beam framework, and dynamically populate the data matching rules into the SQL or ETL script using the Jinja2 template engine. Then, transform the preprocessed fields of the data to be mapped to obtain the mapped data.

4. The data acquisition and mapping method as described in claim 1, characterized in that, During the execution of steps S1-S4, errors are detected in real time and automatically corrected.

5. A data acquisition and mapping system for multiple types of databases, characterized in that, To implement the data acquisition and mapping method as described in any one of claims 1-4, the method includes: The data acquisition module is used to collect the data to be mapped from the source database; The field type inference module is used to analyze the field names of the data to be mapped using a large language model, infer the field types, and preprocess the data to be mapped. The matching rule generation module is used to generate data matching rules between the source database and the target database using reinforcement learning optimization algorithms. The data conversion and storage module converts the fields of the preprocessed data to be mapped according to the data matching rules to obtain the mapped data, and stores the mapped data in the target database.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data acquisition and mapping method as described in any one of claims 1-4.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the data acquisition and mapping method as described in any one of claims 1-4.

Images