Heterogeneous data migration method, migration device, equipment and medium

Abstract

The invention provides a heterogeneous data migration method and device, equipment and a medium, and the method comprises the steps: determining a reading path according to a source cluster, and obtaining to-be-migrated source data from the source cluster based on the reading path; determining a required target data mapping rule according to the current data migration scene; constructing an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and mapping a field of the source data to a mapping field of the intermediate mapping model by using the target data mapping rule to obtain converted source data; and loading the converted source data into the target cluster by adopting a batch loading mode so as to complete migration of the source data. By means of the method and device, data migration in various heterogeneous database environments is supported, and the efficiency of data migration synchronization in the heterogeneous cluster environment is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a heterogeneous data migration method, migration device, equipment and medium. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the growth of data in the field of network security and the continuous improvement of business needs, the demand for cross-platform heterogeneous data synchronization and sharing between basic data is increasing.

[0003] Existing heterogeneous cluster data synchronization technologies mainly focus on synchronization between master and slave databases. Synchronization between heterogeneous cluster databases is still difficult and labor-intensive. Data migration often faces problems such as data loss, complex format conversion, and low migration efficiency. Existing technologies mostly rely on manual configuration and fixed rules for data mapping and processing, which makes it difficult to cope with complex and changing data migration needs. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a heterogeneous data migration method, migration device, equipment and medium, which can support data migration in a variety of heterogeneous database environments, provide stable and reliable data migration solutions in different business scenarios, and effectively improve the efficiency of data migration synchronization in heterogeneous cluster environments.

[0005] In a first aspect, an embodiment of the present application provides a method for migrating heterogeneous data, the method comprising:

[0006] Determine a read path according to the source cluster, and obtain source data to be migrated from the source cluster based on the read path;

[0007] Determine the required target data mapping rules based on the current data migration scenario;

[0008] Building an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and mapping the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain converted source data;

[0009] The converted source data is loaded into the target cluster in a batch loading manner to complete the migration of the source data.

[0010] Furthermore, the step of determining the required target data mapping rules according to the current data migration scenario includes:

[0011] Determining a target data mapping mode from a plurality of preset data mapping modes according to the current data migration scenario, and determining an initial data mapping rule based on the target data mapping mode;

[0012] The initial data mapping rule is verified using a pre-trained mapping rule prediction model to determine the target data mapping rule.

[0013] Further, the using of the pre-trained mapping rule prediction model to verify the initial data mapping rule to determine the target data mapping rule includes:

[0014] Performing data mapping on the source data using the initial data mapping rule to obtain mapping data;

[0015] Inputting the source data and the mapping data into the mapping rule prediction model to determine a prediction mapping rule;

[0016] The predicted mapping rule is used to perform rule verification on the initial data mapping rule. If the verification passes, the initial data mapping rule is determined as the target data mapping rule; wherein the rule verification includes data type checking, consistency checking and logic verification.

[0017] Further, mapping the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain the converted source data includes:

[0018] Performing data cleaning and format conversion on the source data to obtain processed source data;

[0019] The fields of the processed source data are mapped to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain the converted source data.

[0020] Furthermore, the step of loading the converted source data into the target cluster in a batch loading manner includes:

[0021] The converted source data is divided into batches based on a preset data batch loading strategy to obtain multiple groups of batch data; wherein the data batch loading strategy includes a data batch division size and a data loading frequency;

[0022] The multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency.

[0023] Furthermore, when the multiple groups of batch data are loaded into the target cluster in batches according to the data loading frequency, the migration method further includes:

[0024] Using preset monitoring thresholds to detect the batch loading speed of multiple groups of batch data;

[0025] When an abnormal situation is detected, the data batch division size and the data loading frequency in the data batch loading strategy are adjusted according to a preset processing strategy.

[0026] Furthermore, when the multiple groups of batch data are loaded into the target cluster in batches according to the data loading frequency, the migration method further includes:

[0027] Perform data verification on each group of batch data, and record the loading time, data volume and loading results corresponding to each group of batch data; wherein the data verification includes integrity verification, consistency verification and accuracy verification.

[0028] In a second aspect, an embodiment of the present application further provides a heterogeneous data migration device, the migration device comprising:

[0029] A data acquisition module, used to determine a read path according to a source cluster, and acquire source data to be migrated from the source cluster based on the read path;

[0030] A mapping rule determination module is used to determine the required target data mapping rules according to the current data migration scenario;

[0031] A field mapping module, used to construct an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and map the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain converted source data;

[0032] The data migration module is used to load the converted source data into the target cluster in a batch loading manner to complete the migration of the source data.

[0033] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the heterogeneous data migration method as described above are performed.

[0034] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the heterogeneous data migration method as described above are executed.

[0035] The embodiments of the present application provide a heterogeneous data migration method, migration device, equipment and medium. First, a read path is determined according to a source cluster, and source data to be migrated is obtained from the source cluster based on the read path; then, the required target data mapping rules are determined according to the current data migration scenario; an intermediate mapping model is constructed based on the data type of the source cluster and the data type of the target cluster, and the fields of the source data are mapped to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain the converted source data; finally, the converted source data is loaded into the target cluster in a batch loading manner to complete the migration of the source data.

[0036] The migration method provided by the embodiment of the present application can support data migration in a variety of heterogeneous database environments, can provide stable and reliable data migration solutions in different business scenarios, effectively improve the efficiency of data migration synchronization in heterogeneous cluster environments, reduce system maintenance costs, and improve the scalability and fault tolerance of the system. In addition, this solution provides a unified interface through an intermediate mapping model, simplifies the mapping complexity between different data sources and targets, and reduces the need for direct mapping between source data and multiple heterogeneous target clusters. The intermediate mapping model is used to migrate and synchronize source cluster data. When the target database structure changes, it is only necessary to adjust the mapping from the intermediate model to the target database without redoing the entire data source to target mapping, thereby improving the adaptability of data migration synchronization. The intermediate mapping model performs automated data preprocessing to improve the efficiency of data migration, has better error handling and logging mechanisms, ensures the transparency and traceability of the entire migration process, and makes the data migration process more modular and easy to manage and maintain.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a heterogeneous data migration method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a heterogeneous data migration device provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0043] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of data migration technology.

[0044] With the rapid development of big data and cloud computing technologies, the growth of data in the field of network security and the continuous improvement of business needs, the demand for cross-platform heterogeneous data synchronization and sharing between basic data is increasing.

[0045] Research has found that existing heterogeneous cluster data synchronization technologies mainly focus on synchronization between master and slave databases, and synchronization between heterogeneous cluster databases is still difficult and labor-intensive. Data migration often faces problems such as data loss, complex format conversion, and low migration efficiency. Existing technologies mostly rely on manual configuration and fixed rules for data mapping and processing, which makes it difficult to cope with complex and changing data migration needs.

[0046] Based on this, the embodiment of the present application provides a heterogeneous data migration method that can support data migration in multiple heterogeneous database environments, can provide stable and reliable data migration solutions in different business scenarios, and effectively improve the efficiency of data migration synchronization in heterogeneous cluster environments.

[0047] See also Figure 1 , Figure 1 A flow chart of a heterogeneous data migration method provided in an embodiment of the present application. Figure 1 As shown in , the migration method provided by the embodiment of the present application includes:

[0048] S101 . Determine a read path according to a source cluster, and acquire source data to be migrated from the source cluster based on the read path.

[0049] Here, the source cluster refers to the server cluster where the data or application is initially located during data migration or system upgrade. The source data is the data stored in the source cluster and will be migrated to the target cluster later. The target cluster refers to the server cluster where the data or application is ultimately migrated during data migration or system upgrade.

[0050] In the above step S101, the data of the heterogeneous clusters is mainly collected and read. For the above step S101, in the specific implementation, the reading path is determined according to the source cluster, and the source data to be migrated is obtained from the source cluster based on the reading path. Specifically, according to the different reading paths, the storage location of the source cluster data is determined, and the data receiving point of the target cluster is identified to ensure that the data can be correctly transmitted and stored. At the same time, the window function and the corresponding database driver are used to establish a connection with the source cluster to ensure that there is appropriate control authority to access the source data. Then, the data structure of the source cluster is identified, the data type and actual field value of each field in the source data are analyzed, and the size, complexity and update frequency of the data are evaluated. Appropriate pre-processing logic such as data conversion and standardization is designed to ensure the validity and accuracy of the data during the synchronization process. In this way, according to the reading paths of different clusters, it is divided into source cluster data reading and target cluster data reading, and the window function and driver are used to establish a connection with the source data, and the metadata of the source data, such as data structure, data type, field value, etc., are obtained. The acquisition of each source data interface is realized to form a data set to be converted.

[0051] Here, we take the migration and synchronization of heterogeneous data of cyberspace assets as an example. In cyberspace asset management, different types of assets (such as servers, routers, switches, etc.) store their related basic attributes, configuration information, survival status and other data in multiple heterogeneous databases. These heterogeneous databases usually use different data models and formats to store data for different business needs. In the above steps, the data collection and preprocessing stage is first performed. First, the original data such as IP address, asset type, location, open port, service, survival status, etc. in different network asset databases are obtained through API or database connection, and preprocessing operations such as data deduplication, data format unification, and missing value processing are performed. Secondly, while collecting data, determine the data receiving format of the target database cluster, use window functions to process the data in blocks, and improve the efficiency of data collection and transmission through parallel processing and distributed computing technology to ensure the rapid collection and processing of large-scale data.

[0052] S102: Determine required target data mapping rules according to the current data migration scenario.

[0053] Regarding the above step S102, during specific implementation, the data mapping rules required for migrating the source data are determined according to the current data migration scenario.

[0054] As an optional embodiment, with respect to the above step S102, the step of determining the required target data mapping rule according to the current data migration scenario includes:

[0055] Step 1021: Determine a target data mapping mode from a plurality of preset data mapping modes according to the current data migration scenario, and determine an initial data mapping rule based on the target data mapping mode.

[0056] Here, firstly, different data mapping modes, namely preset data mapping modes, are designed according to the requirements of the existing migration scenarios. As an example, the preset data mapping modes include direct mapping, rule mapping, and custom mapping. At the same time, the mapping rules and field type conversion rules of each preset data mapping mode are defined.

[0057] Regarding the above step 1021, in specific implementation, a target data mapping mode is determined from a plurality of preset data mapping modes preset in advance according to the current data migration scenario, and an initial data mapping rule is determined based on the target data mapping mode.

[0058] Step 1022: Verify the initial data mapping rule using a pre-trained mapping rule prediction model to determine the target data mapping rule.

[0059] Here, the mapping rule prediction model adopts a decision tree machine learning model.

[0060] Regarding the above step 1022, during specific implementation, the initial data mapping rule determined in step 1021 is verified using a pre-trained mapping rule prediction model to determine the target data mapping rule.

[0061] Specifically, with respect to the above step 1022, the use of the pre-trained mapping rule prediction model to verify the initial data mapping rule to determine the target data mapping rule includes:

[0062] Step 10221: Use the initial data mapping rule to perform data mapping on the source data to obtain mapping data.

[0063] In the specific implementation of 10221, the source data is mapped using the initial data mapping rule to obtain the mapped data. Specifically, assuming that the source database field set is S = {s1, s2, ..., s m}, the target database field set is T = {t1, t2, ..., t n}, the initial data mapping rule is defined as M: S→T, where M(s i ) = t j, each source field is mapped to the target field The source data is mapped using the initial data mapping rule M to obtain mapped data.

[0064] Step 10222: input the source data and the mapping data into the mapping rule prediction model to determine the prediction mapping rule.

[0065] Here, historical migration data and existing mapping rules are collected to construct a training data set, and the data set is used to train the mapping rule prediction model. At the same time, the validation data set is used to evaluate the trained model to ensure the accuracy and generalization ability of the model, so that the mapping rule prediction model can determine the corresponding mapping rules based on the migration data.

[0066] Regarding the above step 10222, during the specific implementation, the source data and the mapping data are input into the mapping rule prediction model to determine the prediction mapping rules.

[0067] Step 10223: Use the predicted mapping rule to perform rule verification on the initial data mapping rule. If the verification passes, determine the initial data mapping rule as the target data mapping rule.

[0068] Here, the rule verification includes data type checking, consistency checking and logic verification.

[0069] For the above step 10223, the initial data mapping rules are verified using the predicted mapping rules predicted by the mapping rule prediction model. Here, the rule verification includes data type check, consistency check and logic verification. When the verification passes, the initial data mapping rules are determined as the target data mapping rules. Finally, the field mapping relationship is confirmed and applied to the data migration process to ensure the accurate mapping of the data fields. If the verification fails, a prompt message can be sent to the user to enable the user to redetermine the initial data mapping rules, or return to execute step S102 to redetermine the initial data mapping rules, and then continue to execute the steps in step 1021 to verify the initial data mapping rules until the verification passes and the target data mapping rules are determined.

[0070] In this way, different data mapping modes are designed according to the needs of existing migration scenarios, and machine learning models are used to automatically identify and map source data and target data fields, reducing manual intervention and improving the accuracy of data mapping.

[0071] Here, the embodiment of the migration and synchronization processing of heterogeneous data of cyberspace assets in the above steps is continued, and data field mapping is performed in step S102. At this stage, the metadata model of the data in different network asset databases is analyzed and established. First, the metadata model of the data field name, data type, data length, etc. of the source database and the target database is established, and the mapping rules are designed. Secondly, according to business needs and data characteristics, different mapping modes such as one-to-one mapping, many-to-one mapping, one-to-many mapping, etc. are defined, and corresponding data processing rules are designed for each mapping mode.

[0072] S103, constructing an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and mapping the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain converted source data.

[0073] For the above step S103, in the specific implementation, an intermediate mapping model is constructed according to the data type of the data in the source cluster and the data type of the data in the target cluster, and the target data mapping rule determined in step S102 is used to map the fields of the source data to the mapping fields of the intermediate mapping model to obtain the converted source data.

[0074] Specifically, an intermediate mapping model is constructed to map the data types of the source cluster and the target cluster to the common type set of the intermediate mapping model. At the same time, a metadata management module is designed to use a fuzzy matching algorithm to handle situations where the names are different but the semantics are similar. A mapping rule engine is designed to create mapping rules for matching metadata pairs. A transformation processing module is designed to rebuild the converted metadata structure in the target database and perform actual operations such as data cleaning, standardization, and format conversion. A data monitoring module is designed to record detailed information during the data migration process, including performance indicators, errors, and exception handling.

[0075] In the above step S103, data model conversion is mainly implemented. Suppose the field set of the intermediate mapping model is I = {i1, i2, ..., i p}, the transformation rule from source data to the intermediate mapping model is M S :S→I, the conversion rule from target data to the intermediate mapping model is M T :T→I. This process not only involves the construction of mapping models, metadata management, and data type conversion, but also includes data cleaning and the processing of abnormal data.

[0076] As an optional embodiment, with respect to the above step S103, mapping the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain the converted source data includes:

[0077] Step 1031, performing data cleaning and format conversion on the source data to obtain processed source data.

[0078] Regarding the above step 1031, during specific implementation, the conversion processing module in the intermediate mapping model performs necessary cleaning and format conversion on the source data to remove invalid, erroneous or incomplete data and obtain processed source data.

[0079] Step 1032: Map the fields of the processed source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain the converted source data.

[0080] Regarding the above step 1032, in specific implementation, the fields of the processed source data are mapped to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain the converted source data.

[0081] As an optional embodiment, the data monitoring module in the intermediate mapping model can also be used to record detailed information of the data migration and conversion process and throw out abnormal data in a timely manner.

[0082] In this way, by building an intermediate mapping model, the data to be migrated is extracted from the source cluster, the metadata of the source data is analyzed and matched with the metadata management layer of the intermediate model, and the predefined mapping rules are applied to map the fields of the source data to the corresponding fields of the intermediate model. Finally, the source data is cleaned, format converted, and monitored for migration as necessary.

[0083] Here, the embodiment of the migration and synchronization processing of heterogeneous data of cyberspace assets in the above steps is continued. In step S103, the data model conversion stage is performed to build an intermediate mapping model to bridge the differences between the source data and the target data model. Various data formats are analyzed (such as tables in relational databases, documents or key-value pairs in NoSQL databases), and mapping rules are formulated for key fields in each format (such as IP addresses, asset types, locations, survival status, services, etc.). Strategies for handling data anomalies and missing values ​​are defined, such as filling default values ​​for missing values, or excluding records with incorrect formats. Data conversion methods, such as standardization, field mapping, and type conversion, are applied to achieve correct data migration, and the integrity and consistency of data migration are ensured through an incremental synchronization mechanism.

[0084] S104: Load the converted source data into the target cluster in a batch loading manner to complete the migration of the source data.

[0085] For the above step S104, in the specific implementation, the converted source data is loaded into the target cluster by batch loading to complete the migration of the source data from the source cluster to the target cluster. In this way, efficient data loading strategies such as batch insertion and parallel loading are adopted to improve the data synchronization speed and batch import the converted and mapped data into the target database cluster.

[0086] As an optional embodiment, with respect to the above step S104, the step of loading the converted source data into the target cluster in a batch loading manner includes:

[0087] Step 1041 , dividing the converted source data into batches based on a preset data batch loading strategy to obtain multiple groups of batch data.

[0088] Among them, the data batch loading strategy includes data batch division size and data loading frequency.

[0089] For the above step 1041, in the specific implementation, the converted source data is batched based on the data batch size in the preset data batch loading strategy to obtain multiple groups of batch data. Specifically, according to the amount of migrated data, data type, network bandwidth and other system resources, the converted source data is divided into multiple batches and the initial batch size is set, and the frequency of data loading synchronization is set to improve data loading efficiency.

[0090] Step 1042: Batch load multiple groups of batch data into the target cluster according to the data loading frequency.

[0091] Regarding the above step 1042, during specific implementation, multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency in the data batch loading strategy.

[0092] As an optional embodiment, with respect to the above step 1042, when the multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency, the migration method further includes:

[0093] A: Use preset monitoring thresholds to detect the batch loading speed of multiple groups of batch data.

[0094] B: When an abnormal situation is detected, the data batch division size and data loading frequency in the data batch loading strategy are adjusted according to a preset processing strategy.

[0095] For the above steps A-B, in the specific implementation, set a preset monitoring threshold, monitor the consumption of system resources (such as CPU, memory, disk IO) and network bandwidth during data loading, and detect the batch loading speed of multiple groups of batch data in real time. When an abnormal situation is detected, an alarm is automatically issued, and the data batch division size and data loading frequency in the data batch loading strategy are adjusted according to the predefined preset processing strategy. In this way, the batch size and loading frequency are dynamically adjusted using intelligent algorithms, the data loading strategy is optimized, and the system resource consumption and data loading speed during the data loading process are monitored in real time, so as to achieve efficient data synchronization of heterogeneous clusters.

[0096] As an optional embodiment, with respect to the above step 1042, when the multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency, the migration method further includes:

[0097] Perform data verification on each group of batch data, and record the loading time, data volume and loading results corresponding to each group of batch data.

[0098] Here, the data verification includes integrity verification, consistency verification and accuracy verification.

[0099] For the above steps, in the specific implementation, when loading data in batches, perform data verification on each group of batch data, and record the loading time, data volume and loading results corresponding to each group of batch data. If necessary, load the converted source data in batches to the target cluster according to the designed batch loading strategy, verify the integrity, consistency, accuracy, etc. of each batch of loaded data, record the logs of each data loading time, data volume, results, etc., to ensure the consistency of source data and target data.

[0100] Here, the embodiment of the migration and synchronization processing of heterogeneous data of cyberspace assets in the above steps is continued, and the data loading synchronization stage is performed in the above step S104. An efficient data loading strategy, such as batch insertion and parallel loading, is adopted to improve the data synchronization speed, and the converted and mapped data is imported into the target database cluster in batches. After the data is loaded, integrity verification such as data row number verification, field verification, and data content verification is performed to ensure the successful loading of all data and the effectiveness and accuracy of data synchronization.

[0101] The overall solution is described by taking the migration and synchronization of cross-platform network threat intelligence data as another embodiment: the network threat intelligence database usually contains information about network attacks, vulnerabilities, malware, etc. The purpose of this solution is to synchronize the threat intelligence databases between different platforms (for example, open source intelligence platforms and commercial security analysis tools) to improve the efficiency of threat identification and response. The specific process is as follows:

[0102] (1) In the data reading stage, different data reading strategies are designed according to the interface characteristics of different platforms. Use a dedicated driver or API to connect to the security platform, extract security data such as attack type, affected system, attacker IP, etc. Use window functions and dedicated database drivers to establish a connection with the source database, and obtain metadata such as the table structure, data type, field name and value of the source data;

[0103] (2) In the data field mapping stage. First, the metadata sets of the source data and target data clusters are modeled, for example, the "attacker IP" field in the source data set is mapped to the "Source IP" field in the target database. Different data mapping modes are designed according to the requirements of the migration scenario, and field mismatches that occur during the mapping process are processed and converted, such as unifying the IP address fields of different platforms into a standard format and classifying the attack type fields.

[0104] (3) In the data model conversion phase. Construct an intermediate mapping model between network threat intelligence on different platforms and target systems, deconstruct the threat intelligence components of each data source, such as IOC (indicators), TTPs (tactics, techniques, and procedures); consider the coverage and characteristics of different data sources, identify and store key metadata fields (such as attacker IP, domain name, URL, hash value, attack method, severity level, etc.); define and manage data conversion rules from source to target, and handle representation differences of the same information in different data sources; apply mapping rules to perform data conversion, handle field mapping, and format standardization; and keep detailed records of any errors or exceptions that occur during the processing process.

[0105] (4) In the data loading and synchronization phase, the converted threat intelligence data is batch loaded into the target database through data services. At the same time, the business system is put into operation on the target cluster to verify the integrity and accuracy of the migration data synchronization. During this period, the system resource consumption, data loading speed, error rate and other indicators in the loading process are continuously monitored to ensure the smooth completion of the data synchronization task.

[0106] The heterogeneous data migration method provided in the embodiment of the present application first determines a read path according to a source cluster, and obtains source data to be migrated from the source cluster based on the read path; then, determines the required target data mapping rules according to the current data migration scenario; builds an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and uses the target data mapping rules to map the fields of the source data to the mapping fields of the intermediate mapping model to obtain the converted source data; finally, loads the converted source data into the target cluster in a batch loading manner to complete the migration of the source data.

[0107] The migration method provided by the embodiment of the present application can support data migration in a variety of heterogeneous database environments, can provide stable and reliable data migration solutions in different business scenarios, effectively improve the efficiency of data migration synchronization in heterogeneous cluster environments, reduce system maintenance costs, and improve the scalability and fault tolerance of the system. In addition, this solution provides a unified interface through an intermediate mapping model, simplifies the mapping complexity between different data sources and targets, and reduces the need for direct mapping between source data and multiple heterogeneous target clusters. The intermediate mapping model is used to migrate and synchronize source cluster data. When the target database structure changes, it is only necessary to adjust the mapping from the intermediate model to the target database without redoing the entire data source to target mapping, thereby improving the adaptability of data migration synchronization. The intermediate mapping model performs automated data preprocessing to improve the efficiency of data migration, has better error handling and logging mechanisms, ensures the transparency and traceability of the entire migration process, and makes the data migration process more modular and easy to manage and maintain.

[0108] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a heterogeneous data migration device provided in an embodiment of the present application. Figure 2 As shown in , the migration device 200 includes:

[0109] The data acquisition module 201 is used to determine a read path according to a source cluster, and acquire source data to be migrated from the source cluster based on the read path;

[0110] A mapping rule determination module 202 is used to determine the required target data mapping rule according to the current data migration scenario;

[0111] A field mapping module 203 is used to construct an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and map the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain converted source data;

[0112] The data migration module 204 is used to load the converted source data into the target cluster in a batch loading manner to complete the migration of the source data.

[0113] Furthermore, when the mapping rule determination module 202 is used to determine the required target data mapping rule according to the current data migration scenario, the mapping rule determination module 202 is also used to:

[0114] Determining a target data mapping mode from a plurality of preset data mapping modes according to the current data migration scenario, and determining an initial data mapping rule based on the target data mapping mode;

[0115] The initial data mapping rule is verified using a pre-trained mapping rule prediction model to determine the target data mapping rule.

[0116] Furthermore, when the mapping rule determination module 202 is used to verify the initial data mapping rule using the pre-trained mapping rule prediction model to determine the target data mapping rule, the mapping rule determination module 202 is also used to:

[0117] Performing data mapping on the source data using the initial data mapping rule to obtain mapping data;

[0118] Inputting the source data and the mapping data into the mapping rule prediction model to determine a prediction mapping rule;

[0119] The predicted mapping rule is used to perform rule verification on the initial data mapping rule. If the verification passes, the initial data mapping rule is determined as the target data mapping rule; wherein the rule verification includes data type checking, consistency checking and logic verification.

[0120] Furthermore, when the field mapping module 203 is used to map the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain the converted source data, the field mapping module 203 is also used to:

[0121] Performing data cleaning and format conversion on the source data to obtain processed source data;

[0122] The fields of the processed source data are mapped to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain the converted source data.

[0123] Furthermore, when the data migration module 204 is used to load the converted source data into the target cluster in a batch loading manner, the data migration module 204 is also used to:

[0124] The converted source data is divided into batches based on a preset data batch loading strategy to obtain multiple groups of batch data; wherein the data batch loading strategy includes a data batch division size and a data loading frequency;

[0125] The multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency.

[0126] Furthermore, when the data migration module 204 is used to batch load multiple groups of batch data into the target cluster according to the data loading frequency, the data migration module 204 is also used to:

[0127] Using preset monitoring thresholds to detect the batch loading speed of multiple groups of batch data;

[0128] When an abnormal situation is detected, the data batch division size and the data loading frequency in the data batch loading strategy are adjusted according to a preset processing strategy.

[0129] Furthermore, when the data migration module 204 is used to batch load multiple groups of batch data into the target cluster according to the data loading frequency, the data migration module 204 is also used to:

[0130] Perform data verification on each group of batch data, and record the loading time, data volume and loading results corresponding to each group of batch data; wherein the data verification includes integrity verification, consistency verification and accuracy verification.

[0131] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0132] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 via the bus 330. When the machine-readable instructions are executed by the processor 310, the above-mentioned Figure 1 The steps of the heterogeneous data migration method in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0133] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the heterogeneous data migration method in the method embodiment shown in the figure, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for migrating heterogeneous data. It is characterized in that The migration method comprises: Determine a read path according to the source cluster, and obtain source data to be migrated from the source cluster based on the read path; Determine the required target data mapping rules based on the current data migration scenario; Building an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and mapping the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rule to obtain converted source data; The converted source data is loaded into the target cluster in a batch loading manner to complete the migration of the source data.

2. The migration method according to claim 1, It is characterized in that Determining the required target data mapping rules according to the current data migration scenario includes: Determining a target data mapping mode from a plurality of preset data mapping modes according to the current data migration scenario, and determining an initial data mapping rule based on the target data mapping mode; The initial data mapping rule is verified using a pre-trained mapping rule prediction model to determine the target data mapping rule.

3. The migration method according to claim 2, It is characterized in that The using a pre-trained mapping rule prediction model to verify the initial data mapping rule to determine the target data mapping rule includes: Performing data mapping on the source data using the initial data mapping rule to obtain mapping data; Inputting the source data and the mapping data into the mapping rule prediction model to determine a prediction mapping rule; The predicted mapping rule is used to perform rule verification on the initial data mapping rule. If the verification passes, the initial data mapping rule is determined as the target data mapping rule; wherein the rule verification includes data type checking, consistency checking and logic verification.

4. The migration method according to claim 1, It is characterized in that The using the target data mapping rule to map the fields of the source data to the mapping fields of the intermediate mapping model to obtain the converted source data includes: Performing data cleaning and format conversion on the source data to obtain processed source data; The fields of the processed source data are mapped to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain the converted source data.

5. The migration method according to claim 1, It is characterized in that The step of loading the converted source data into the target cluster in a batch loading manner includes: The converted source data is divided into batches based on a preset data batch loading strategy to obtain multiple groups of batch data; wherein the data batch loading strategy includes a data batch division size and a data loading frequency; The multiple groups of batch data are batch loaded into the target cluster according to the data loading frequency.

6. The migration method according to claim 5, It is characterized in that When the multiple groups of batch data are loaded into the target cluster in batches according to the data loading frequency, the migration method further includes: Using preset monitoring thresholds to detect the batch loading speed of multiple groups of batch data; When an abnormal situation is detected, the data batch division size and the data loading frequency in the data batch loading strategy are adjusted according to a preset processing strategy.

7. The migration method according to claim 5, It is characterized in that When the multiple groups of batch data are loaded into the target cluster in batches according to the data loading frequency, the migration method further includes: Perform data verification on each group of batch data, and record the loading time, data volume and loading results corresponding to each group of batch data; wherein the data verification includes integrity verification, consistency verification and accuracy verification.

8. A heterogeneous data migration device, It is characterized in that The migration device comprises: A data acquisition module, used to determine a read path according to a source cluster, and acquire source data to be migrated from the source cluster based on the read path; A mapping rule determination module is used to determine the required target data mapping rules according to the current data migration scenario; A field mapping module, used to construct an intermediate mapping model based on the data type of the source cluster and the data type of the target cluster, and map the fields of the source data to the mapping fields of the intermediate mapping model using the target data mapping rules to obtain converted source data; The data migration module is used to load the converted source data into the target cluster in a batch loading manner to complete the migration of the source data.

9. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the heterogeneous data migration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the heterogeneous data migration method according to any one of claims 1 to 7 are executed.

Images