A data migration method and device, electronic equipment and storage medium

Abstract

The application discloses a data migration method and device, electronic equipment and storage medium. The method comprises the following steps: constructing a hierarchical node graph according to demand elements of a data migration system under multiple demand types; different demand types are located in different levels; obtaining a standard data model matched with at least one data migration system respectively, the standard data model is used for storing the mapping relationship between multiple elements of the data migration system, and two elements in each element mapping relationship belong to different demand types; establishing the mapping relationship between demand elements in adjacent levels in the hierarchical node graph according to the standard data model, forming a migration mapping relationship graph; and migrating data in each data migration system to the data migration system according to the migration mapping relationship graph. Through the technical scheme, the data migration process based on data assets can be automated, the workload of manual data migration is reduced, and the data migration efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data migration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data assets refer to data resources, recorded physically or electronically, owned or controlled by individuals or enterprises, that can bring future economic benefits to the enterprise. Simultaneously, data assets are datasets in cyberspace that possess ownership, value, measurability, and readability. With rapid economic development and the booming growth of e-commerce, existing information systems often cannot meet all customer business needs. Many enterprises have made varying degrees of customizations to their existing systems, leading to a demand for system upgrades. However, system upgrades require the migration of historical data from the existing system, which is no easy task.

[0003] When updating existing systems, the mapping relationship between the target field and the source field is usually established manually, the migration script between databases is developed manually, and the data quality is checked manually during the migration process.

[0004] In the process of developing this invention, the inventors discovered the following shortcomings in the existing technology: Existing data migration technologies primarily address the ease of script writing, assuming that the data relationship between the migrated-out and migrated-in data already exists, and that the migration device automatically generates the script by configuring the mapping relationship. However, generating the data relationship between the migrated-out and migrated-in data depends on understanding the distribution of the migrated-in data within the migrated-out system, requiring extensive business, system, and data research. This results in a large workload for manual data migration and low data migration efficiency. Summary of the Invention

[0005] This invention provides a data migration method, apparatus, electronic device, and storage medium to solve the problems of high workload and low efficiency in manual data migration in existing data migration processes.

[0006] In a first aspect, embodiments of the present invention provide a data migration method, the method comprising:

[0007] Based on the requirement elements of the data migration system under multiple requirement types, a hierarchical node diagram is constructed. The requirement types include data source, entity, and business attribute; different requirement types are located at different levels.

[0008] Obtain a standard data model that matches at least one data migration system. The standard data model is used to store the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different requirement types.

[0009] Based on the standard data models, a mapping relationship between each requirement element in adjacent levels is established in the hierarchical node graph to form a migration mapping relationship graph.

[0010] According to the migration mapping diagram, the data from each of the data migration-out systems is migrated to the data migration-in system.

[0011] In a second aspect, embodiments of the present invention provide a data migration apparatus, the apparatus comprising:

[0012] The hierarchical construction module is used to build a hierarchical node diagram based on the requirement elements of the data migration system under multiple requirement types. The requirement types include data sources, entities, and business attributes; different requirement types are located at different levels.

[0013] The standard data model matching module is used to obtain the standard data model that matches at least one data migration system. The standard data model stores the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different demand types.

[0014] The migration mapping graph construction module is used to establish the mapping relationship between each requirement element in the adjacent levels of the hierarchical node graph according to each standard data model, thus forming a migration mapping graph.

[0015] The data migration module migrates data from each of the data migration-out systems to the data migration-in system according to the migration mapping diagram.

[0016] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data migration method according to any embodiment of the present invention.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the data migration method described in any embodiment of the present invention.

[0021] The technical solution of this invention constructs a hierarchical node graph based on the required elements and obtains standard data models that match at least one data migration system. Then, a migration mapping relationship graph is obtained based on the standard data models and the hierarchical node graph. Finally, according to the migration mapping relationship graph, the data in each data migration system is migrated to the data migration system. By using the existing mapping relationship between elements in the data migration system, the migration mapping relationship required for the data migration process is adaptively established. This solves the problems of large workload and low efficiency of manual data migration during the data migration process, realizes the automation of the data migration process based on data assets, reduces the workload of manual data migration, and improves the efficiency of data migration.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1a This is a schematic diagram of a data migration structure implemented using existing technology;

[0025] Figure 1b This is a flowchart of a data migration method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a data migration method provided according to Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a data migration device according to Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data migration method of this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] To enable those skilled in the art to better understand the present invention, a brief introduction will first be given to the complete processing procedure of vertex data in the prior art. Specifically, as follows... Figure 1a As shown:

[0032] Current data migration technologies primarily address the ease of script writing, assuming the data relationships between the migrating-out and migrating-in systems already exist. The mapping relationships are configured within the data migration device to automatically generate scripts. However, generating these relationships relies on understanding the distribution of the migrating-in data within the migrating-out system, requiring extensive business, system, and data research, and is inherently subject to human subjectivity. Specifically, for example… Figure 1a As shown: For n databases in the system that need to be migrated, manual business research, system research, and data research are conducted to establish a mapping relationship with the data in the database to be migrated. This process involves a large amount of work.

[0033] After the above process, the mapping relationship between the database of the system being migrated in and the database of the system being migrated out is obtained. The existing technology is to carry out script automation work on the basis of the above operation, which only reduces the workload of script development, but the formation of the mapping relationship relies entirely on manual sorting.

[0034] This invention relates to data migration based on data assets, which may include enterprise-level data models, metadata assets, and data lineage. A hierarchical node diagram is constructed based on the required elements, and standard data models matching at least one data migration-out system are obtained. A migration mapping diagram is then derived based on the standard data models and the hierarchical node diagram. Finally, data from each data migration-out system is migrated to the data migration-in system according to the migration mapping diagram. This invention solves the problems of high workload and low efficiency in manual data migration, automating the data migration process based on data assets, reducing the workload of manual data migration, and improving the efficiency of data migration.

[0035] Example 1

[0036] Figure 1b This is a flowchart of a data migration method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where data assets need to be migrated between different database systems. The method can be executed by a data migration device, which can be implemented in hardware and / or software. This data migration device can be configured in a data asset device with data migration functionality. Figure 1b As shown, the method includes:

[0037] S110. Based on the requirement elements of the data migration system under multiple requirement types, construct a hierarchical node diagram.

[0038] The types of requirements include data sources, entities, and business attributes; different types of requirements are located at different levels.

[0039] The data import system is a manually designed system with a specific function and data storage capabilities. It is easy to understand that, because the data import system has a specific function, the required elements can be defined through that function. For example, taking a bank savings card system as an example, if the bank savings card system is set as the data import system, it is easy to understand that this system has bank savings card functionality, and the required elements may include customer name, savings card type, customer contact information, etc.

[0040] The requirement element is a specific category of the requirement type. A requirement type can contain multiple requirement elements, but the same requirement element can only belong to one requirement type.

[0041] Optionally, based on the requirement elements of the data migration system under multiple requirement types, a hierarchical node diagram can be constructed, including:

[0042] Based on all the required elements of the data migration system under the data source type, construct the first-level node;

[0043] Based on all the required elements of the data migration system under the entity type, construct the second-level node;

[0044] Based on all the required elements of the data migration system under the business attribute type, a third-level node is constructed to obtain the aforementioned hierarchical node diagram.

[0045] In this embodiment, taking a bank savings card system as an example, the requirement elements under the data source type may include: customer elements, product elements, and data elements, etc. The types of requirement elements are determined by the system preset. All the requirement elements under the data source type constitute the first-level nodes, and each requirement element under a different data source type can be a node in the first level. Similarly, the requirement elements under the entity type in the data migration system may include: customer relationships, customer contact information, and customer credit rating, etc. All the requirement elements under the entity type constitute the second-level nodes, and each requirement element under a different entity type can be a node in the second level. Furthermore, the requirement elements under the business attribute type may include: customer phone number, customer email address, and product number, etc. All the requirement elements under the business attribute type constitute the third-level nodes, and each requirement element under a different business attribute type can be a node in the third level.

[0046] Furthermore, the hierarchical node graph is composed of a first-level node, a second-level node, and a third-level node; wherein, the first-level node is the highest level of the hierarchical node graph and is adjacent to the second-level node, and the third-level node is the lowest level of the hierarchical node graph and is adjacent to the second-level node.

[0047] S120. Obtain a standard data model that matches at least one data migration system.

[0048] In the standard data model, multiple elements are used to store the mapping relationships between the data migration system; furthermore, the two elements in each mapping relationship belong to different requirement types.

[0049] In this embodiment, a requirement element under a data source type can establish a mapping relationship with a requirement element under at least one entity type, and a requirement element under an entity type can establish a mapping relationship with a requirement element under at least one business attribute type.

[0050] For example, taking the requirement elements exemplified in step S110, requirement elements under the data source type may include: customer-type elements, product-type elements, and data-type elements, etc. Requirement elements under the entity type may include: customer relationships, customer contact information, and customer credit rating, etc. Requirement elements under the business attribute type may include: customer phone number, customer email address, and product number, etc. It should be clear to those skilled in the art that there is a mapping relationship between customer-type elements and customer relationships, and between customer-type elements and customer contact information; similarly, there is a mapping relationship between customer contact information and customer phone number, and between customer contact information and customer email address.

[0051] S130. Based on each standard data model, establish the mapping relationship between each requirement element in the adjacent levels in the hierarchical node diagram to form a migration mapping relationship diagram.

[0052] The mapping relationship can be a mapping relationship between nodes in two adjacent levels. Typically, a node in a higher level can correspond to at least one adjacent node in a lower level.

[0053] The migration mapping graph includes the hierarchical relationship of each level node in the hierarchical node graph, and the mapping relationship between nodes in different levels.

[0054] S140. According to the migration mapping diagram, migrate the data from each of the data migration-out systems to the data migration-in system.

[0055] The data asset device can be configured to link the data sources in the data migration system and the data migration system, as well as the operating environment of the system. After the configuration is completed, the data migration operation can be performed automatically.

[0056] Optionally, a migration script is generated based on the migration mapping graph; wherein the migration script includes a migration rule module and a verification rule module;

[0057] The data from the data migration out system is processed by the migration rule module in the migration script and transferred to the data migration in system, whereby the data migration in system obtains the first-level migration result.

[0058] Furthermore, the migration rule module includes node information and mapping relationship information in the migration mapping graph, which enables data to be migrated to the migration module according to the rules in the migration script.

[0059] The verification rules module can be used to verify the correctness of the migrated business attribute data.

[0060] Furthermore, the verification rule module includes: data verification rules and error result output rules; wherein, both the data verification rules and error result output rules can be preset rules, which can be set or changed manually.

[0061] Optionally, the data from the data migration out system is processed and transferred to the data migration in the migration script through the migration rule module. After the migration in the system obtains the first-level migration result, it further includes:

[0062] The first-level migration result is input into the migration script, and the verification rule module in the migration script is used to process and transmit it to the data migration system.

[0063] If the first-level migration result meets the verification criteria of the verification rules module, the migration script will transfer the data to the data migration system, and the data migration system will obtain the second-level migration result.

[0064] If the first-level migration result does not meet the verification criteria of the verification rules module, the migration script outputs an error result set and performs error processing on the error result set.

[0065] The verification rules are set manually and may include non-empty rules and association rules. Furthermore, non-empty rules can filter out blank or garbled invalid data to ensure the correctness of the migration data, while association rules can filter out data that does not conform to the node information format to ensure the validity of the migration data.

[0066] The secondary migration result refers to data that has been verified by the verification rules and is correct in terms of data format and content.

[0067] The error result set consists of data information from the first-level migration results that do not meet the verification criteria of the verification rule module. This data information can be further processed manually or by the system before being re-entered into the system for transmission.

[0068] The solution provided in this embodiment of the invention adds a verification module to the migrated data, which reduces the workload of manual verification, ensures the correctness and effectiveness of data migration, and improves work efficiency.

[0069] The technical solution of this invention constructs a hierarchical node diagram based on the requirement elements and obtains standard data models that match at least one data migration out system. Then, a migration mapping relationship diagram is obtained based on the standard data models and the hierarchical node diagram. Finally, according to the migration mapping relationship diagram, the data in each data migration out system is migrated to the data migration in system and verified. The verified data is then migrated to the data migration in system to complete the data migration. This solves the problems of large workload and low efficiency of manual data migration during the data migration process, realizes the automation of the data migration process based on data assets, reduces the workload of manual data migration, ensures the correctness and effectiveness of data migration, and improves the efficiency of data migration.

[0070] Example 2

[0071] Figure 2 This is a flowchart of a data migration method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiment. In this embodiment, the operation of establishing a mapping relationship between each requirement element in an adjacent level in the hierarchical node diagram according to each standard data model to form a migration mapping relationship diagram is specifically specified as follows: obtaining the current standard data model corresponding to the data migration system being processed; obtaining the current processing level in the hierarchical node diagram in order from low to high level, and obtaining the target requirement type corresponding to the previous level adjacent to the current processing level; searching in the current standard data model according to each requirement element in the current processing level and the target requirement type, and establishing a mapping relationship between each requirement element in the current processing level and the matching previous level requirement element in the hierarchical node diagram according to the search results; returning to execute the operation of obtaining the current processing level in the hierarchical node diagram in order from low to high level, until processing reaches the second level of the hierarchical node diagram.

[0072] Correspondingly, such as Figure 2 As shown, the method includes:

[0073] S210. Based on the requirement elements of the data migration system under multiple requirement types, construct a hierarchical node diagram.

[0074] Specifically, this requirement type can include: data source type, entity type, and business attribute type.

[0075] In this embodiment, the method of constructing a hierarchical node graph may include:

[0076] Based on all the requirement elements of the data migration system under the data source type, construct the first-level node; based on all the requirement elements of the data migration system under the entity type, construct the second-level node; based on all the requirement elements of the data migration system under the business attribute type, construct the third-level node to obtain the hierarchical node diagram.

[0077] S220. Obtain the current standard data model corresponding to the data migration system currently being processed.

[0078] The standard data model includes: mapping relationships between multiple elements of the data migration system, where the two elements in each mapping relationship belong to different demand types.

[0079] S230. In the order from low to high level, obtain the current processing level in the hierarchical node diagram, and obtain the target requirement type corresponding to the previous level adjacent to the current processing level.

[0080] The hierarchical node diagram is composed of a first-level node, a second-level node, and a third-level node; furthermore, the first-level node is the highest level of the hierarchical node diagram and is adjacent to the second-level node, and the third-level node is the lowest level of the hierarchical node diagram and is adjacent to the second-level node.

[0081] In this embodiment, the third level can be obtained first as the current processing level, and the target requirement type, that is, the entity type, corresponding to the second level adjacent to the third level can be obtained.

[0082] S240. Search the current standard data model according to the requirement elements and target requirement types in the current processing level, and establish the mapping relationship between each requirement element in the current processing level and the matching requirement element of the previous level in the hierarchical node graph based on the search results.

[0083] In this embodiment, after determining the current processing level, the mapping relationship of each requirement element in the current processing level can be found in the current standard data model. At the same time, the found mapping relationship must also satisfy that the other requirement element besides the above-mentioned requirement element must belong to the target requirement type.

[0084] For example, taking the customer email address in the third level as the current requirement element, as shown in the example above, the customer email address is a business attribute requirement element, meaning the node containing the customer email address is located in the third level. The requirement element that has a mapping relationship with the customer email address can be the customer contact information, which is an entity type, meaning the node containing the customer contact information is located in the second level. Furthermore, a mapping relationship can be established between the node containing the customer email address in the third level and the node containing the customer contact information in the second level. That is, the mapping relationship between each requirement element in the current processing level and the matching requirement element in the previous level can be established in the hierarchical node graph.

[0085] Optionally, obtain the current requirement element in the current processing level, and obtain at least one element mapping relationship in the current standard data model that corresponds to both the current requirement element and the target requirement type;

[0086] Wherein, if the current requirement element has a mapping relationship with a unique target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matched target requirement type.

[0087] Specifically, the current requirement element is set as a first-level savings account, and the first-level savings account is a requirement element of the business attribute requirement type. This requirement element is set to be able to establish a mapping relationship only with the entity requirement type of "customer level". Then, the mapping established between the second-level node containing the customer level and the third-level node containing the first-level savings account is the mapping relationship between the current requirement element and the matching target requirement type.

[0088] If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry the same data, then the initial target requirement type is selected according to the data lineage, and the mapping relationship between the target requirement type and the current requirement element is used as the mapping relationship between the current requirement element and the matched target requirement type.

[0089] The data lineage refers to a relationship similar to human social kinship that forms between data during the generation, processing, and circulation of data; furthermore, the data lineage records the flow relationship between the original data of each system.

[0090] For example, the current requirement element is set to an ID card number, and the ID card number is a requirement element of the business attribute requirement type. This element can be mapped from two entity requirement types: debit card customer information and credit card customer information. The two contain the same ID card number information, and the ID card number in the credit card customer information is obtained from the debit card customer information. Based on data lineage, the ID card number information in the debit card customer information is determined to be the original information. Furthermore, the mapping established between the second-level node containing the debit card customer information and the third-level node containing the ID card number is the mapping relationship between the current requirement element and the matched target requirement type.

[0091] If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry different data, then the target requirement type that meets the rules is selected according to the preset rules, and the mapping relationship between the target requirement type and the current requirement element is taken as the mapping relationship between the current requirement element and the matched target requirement type.

[0092] For example, the current requirement element is set to a customer email address, and the customer email address is a requirement element of the business attribute requirement type. This element can be mapped from two entity requirement types: debit card customer information and credit card customer information. The customer email information contained in the two is different, and the ID card number in the credit card customer information is obtained from the debit card customer information. Then, according to the judgment rule, it is determined that the email information in the debit card customer information has been modified, and it is determined that the email information in the debit card customer information is the latest information. Furthermore, the mapping established between the second-level node containing the debit card customer information and the third-level node containing the customer email address is the mapping relationship between the current requirement element and the matched target requirement type.

[0093] Optionally, if the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types do not have a data migration relationship, then all target requirement types are merged into the final target requirement type, and the mapping relationship between the target requirement type and the current requirement element is used as the mapping relationship between the current requirement element and the matched target requirement type.

[0094] Specifically, the current requirement element is set as a mobile phone number, and the mobile phone number is a requirement element of the business attribute requirement type. This element can establish a mapping relationship between two entity requirement types: bank counter information and mobile client information. The mobile phone numbers contained in the two are different, and the methods for obtaining the mobile phone numbers are independent of each other, and there is no information flow between them. Then, the mobile phone number information in the second-level node containing bank counter information and the second-level node containing mobile client information are merged to form the final target requirement type.

[0095] Based on the mapping relationship between the at least one element, determine the target upper-level requirement element that matches the current requirement element;

[0096] Establish a mapping relationship between the current requirement element and the requirement element at the next higher level in the hierarchical node diagram;

[0097] Return to the operation that retrieves the current required element in the current processing level, until all required elements in the current processing level have been processed.

[0098] It is easy to understand that once the mapping relationship is determined, since the mapping relationship contains at least two different levels of demand elements, a mapping relationship between the current demand element and the target demand element at the next higher level can be established in the hierarchical node diagram.

[0099] S250. Detect whether the current processing level is the second level of the node graph. If yes, execute S260; otherwise, return to execute S230.

[0100] It is easy to understand that since the current processing level is obtained in the hierarchical node graph in order from low to high, there is no higher level above the first level. That is, the nodes in the first level node graph cannot establish a mapping relationship with the nodes of higher levels. It can be concluded that when the system finishes processing the second level of the hierarchical node graph, that is, when all the nodes in the second level have established a mapping with the nodes in the first level, the hierarchical node graph is finished.

[0101] S260. Determine the currently updated hierarchical node graph as a migration mapping relationship graph.

[0102] S270. According to the migration mapping diagram, migrate the data from each of the data migration-out systems to the data migration-in system.

[0103] The technical solution of this invention constructs a hierarchical node diagram based on demand elements and obtains standard data models that match at least one data migration system. Following a top-down order, the current processing level is obtained from the hierarchical node diagram, along with the target demand type corresponding to the adjacent upper level. Then, based on the target demand type, a mapping relationship is established between each demand element in the current processing level and the matching upper-level demand element in the hierarchical node diagram. Finally, a migration mapping relationship diagram is obtained based on the standard data model and the hierarchical node diagram. According to the migration mapping relationship diagram, data from each data migration system is migrated to the data migration system. This solves the problems of high workload and low efficiency in manual data migration, automating the data migration process based on data assets, reducing the workload of manual data migration, and improving the efficiency of data migration. Detailed Implementation

[0105] To more clearly illustrate the technical solutions provided in the embodiments of the present invention, this embodiment provides a specific implementation method as follows:

[0106] In the specific implementation provided in this embodiment, the migration-in system processes data within a data asset device, clarifying the requirement elements under each different requirement type. The standard data model stores the mapping relationships between multiple elements of the data migration-out system. If a mapping relationship is established between the current requirement element and the target requirement type, the data source range of this requirement element can be directly determined. If the current requirement element has a mapping relationship with only one target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matching target requirement type. If the current requirement element has mapping relationships with multiple target requirement types, data lineage or preset rules can be used to determine which source should be selected. After determining the mapping relationship between the current requirement element of the migration-in system and the matching target requirement type, a data script can be directly generated using the data asset device, and verification rules for each requirement element can be configured in the script, such as "not empty" or "not garbled characters" verification rules. The data asset device can be configured to the data source links and operating environments of each system, and can run automatically after configuration. After the operation is completed, the data that passes verification has completed the data migration, and problematic data will automatically generate an error result set for error handling.

[0107] The implementation steps of a specific embodiment provided in this example are as follows:

[0108] Step 1: Construct a hierarchical node diagram based on the requirement elements of the data migration system under multiple requirement types.

[0109] Step 2: Based on all the requirement elements of the data migration system under the data source type, construct the first-level node; based on all the requirement elements of the data migration system under the entity type, construct the second-level node; based on all the requirement elements of the data migration system under the business attribute type, construct the third-level node to obtain the hierarchical node diagram.

[0110] Step 3: Obtain the current standard data model corresponding to the data migration system being processed.

[0111] Step 4: In the hierarchy node graph, obtain the current processing level and the target requirement type corresponding to the previous level adjacent to the current processing level, in order from the lowest to the highest level.

[0112] Step 5: Search within the current standard data model based on the requirement elements and target requirement types in the current processing level.

[0113] Step 6: Determine whether the current requirement element has a mapping relationship with the only target requirement type; if not, proceed to step 7; if yes, proceed to step 8.

[0114] Step 7: Determine if there is a data migration relationship between multiple target requirement types; if yes, proceed to step 9; otherwise, proceed to step 10.

[0115] Step 8: Establish the mapping relationship between each requirement element in the current processing level and the matching requirement element in the previous level in the hierarchical node diagram, and then execute step 13.

[0116] Step 9: Determine whether the data carried by multiple target requirement types are the same; if yes, proceed to step 11; if no, proceed to step 12.

[0117] Step 10: Merge all target requirement types into the final target requirement type, and use the mapping relationship between the target requirement type and the current requirement element as the mapping relationship between the current requirement element and the matched target requirement type. Return to step 8.

[0118] Step 11: Select the initial target requirement type based on data lineage, and use the mapping relationship between the target requirement type and the current requirement element as the mapping relationship between the current requirement element and the matched target requirement type. Return to step 8.

[0119] Step 12: Filter the target requirement types that meet the preset rules, and use the mapping relationship between the target requirement type and the current requirement element as the mapping relationship between the current requirement element and the matched target requirement type. Then return to step 8.

[0120] Step 13: In order from low to high level, obtain the operation of the current processing level in the hierarchical node graph until the processing reaches the second level of the hierarchical node graph.

[0121] Step 14: Generate a migration mapping relationship graph based on the processed hierarchical node graph.

[0122] Step 15: According to the migration mapping diagram, migrate the data from each data migration-out system to the data migration-in system.

[0123] Step 16: Input the first-level migration results into the migration script, and use the verification rule module in the migration script to process and transfer them to the data migration system.

[0124] Step 17: Determine whether the first-level migration result meets the verification criteria of the verification rules module; if yes, proceed to step 18; otherwise, proceed to step 19.

[0125] Step 18: The migration script transmits the data to the data migration system, and the data migration system obtains the secondary migration result.

[0126] Step 19: The migration script outputs an error result set and performs error handling on the error result set.

[0127] The entire data migration process was carried out within the data asset device, which reduced the reliance of the personnel implementing the migration system on their knowledge of other systems, reduced the workload of manual analysis and design, and improved the efficiency and accuracy of the entire data migration.

[0128] Example 3

[0129] Figure 3 This is a schematic diagram of a data migration device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0130] The hierarchical construction module 310 is used to construct a hierarchical node diagram based on the requirement elements of the data migration system under multiple requirement types. The requirement types include data sources, entities, and business attributes; different requirement types are located at different levels.

[0131] The standard data model matching module 320 is used to obtain a standard data model that matches at least one data migration system. The standard data model is used to store the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different demand types.

[0132] The migration mapping graph construction module 330 is used to establish a mapping relationship between each requirement element in an adjacent level in the hierarchical node graph according to each standard data model, thereby forming a migration mapping graph.

[0133] The data migration module 340 migrates data from each of the data migration-out systems to the data migration-in system according to the migration mapping diagram.

[0134] The technical solution of this invention constructs a hierarchical node diagram based on the requirement elements, obtains a standard data model that matches at least one data migration out system, and then obtains a migration mapping relationship diagram based on the standard data model and the hierarchical node diagram. Finally, according to the migration mapping relationship diagram, the data in each data migration out system is migrated to the data migration in system. This solves the problems of large workload and low efficiency of manual data migration during the data migration process, realizes the automation of the data migration process based on data assets, reduces the workload of manual data migration, and improves the efficiency of data migration.

[0135] Based on the above embodiments, the migration mapping graph construction module 330 may include:

[0136] The current standard data model acquisition unit is used to acquire the current standard data model corresponding to the data migration system currently being processed.

[0137] The search condition acquisition unit is used to obtain the current processing level in the hierarchical node graph in order from low to high level, and to obtain the target requirement type corresponding to the previous level adjacent to the current processing level.

[0138] The mapping relationship lookup unit is used to search in the current standard data model based on each requirement element and target requirement type in the current processing level, and establish a mapping relationship between each requirement element in the current processing level and the matching requirement element in the previous level in the hierarchical node graph based on the search results.

[0139] The first repetitive execution unit is used to return to the execution in order from low to high level, to obtain the operation of the current processing level in the hierarchical node graph, until the processing reaches the second level of the hierarchical node graph.

[0140] Based on the above embodiments, the mapping relationship lookup unit further includes:

[0141] The element mapping relationship acquisition sub-unit is used to acquire the current requirement element in the current processing level, and to acquire at least one element mapping relationship in the current standard data model that corresponds to both the current requirement element and the target requirement type.

[0142] The superior requirement element determination subunit is used to determine the target superior requirement element that matches the current requirement element based on the mapping relationship between the at least one element.

[0143] The mapping relationship establishment sub-unit is used to establish a mapping relationship between the current requirement element and the target requirement element at the next higher level in the hierarchical node diagram;

[0144] The second repetitive execution module is used to return to the operation of retrieving the current demand element in the current processing level, until all demand elements in the current processing level have been processed.

[0145] Based on the above embodiments, the mapping relationship establishes a sub-unit, which can be specifically used for:

[0146] If the current requirement element has a mapping relationship with a unique target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matching target requirement type;

[0147] If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry the same data, then the initial target requirement type is selected according to the data lineage, and the mapping relationship between the target requirement type and the current requirement element is used as the mapping relationship between the current requirement element and the matched target requirement type.

[0148] If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry different data, then the target requirement type that meets the rules is selected according to the preset rules, and the mapping relationship between the target requirement type and the current requirement element is taken as the mapping relationship between the current requirement element and the matched target requirement type.

[0149] Based on the above embodiments, the mapping relationship establishes a sub-unit, which can be specifically used for:

[0150] If the current requirement element has a mapping relationship with a unique target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matching target requirement type;

[0151] If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types do not have a data migration relationship, then all target requirement types are merged into the final target requirement type, and the mapping relationship between the target requirement type and the current requirement element is taken as the mapping relationship between the current requirement element and the matched target requirement type.

[0152] Based on the above embodiments, the data import module 340 can be used for:

[0153] A migration script is generated based on the migration mapping graph; wherein, the migration script includes a migration rule module and a verification rule module;

[0154] The data from the data migration out system is processed by the migration rule module in the migration script and transferred to the data migration in system, whereby the data migration in system obtains the first-level migration result.

[0155] The verification rule module can be used to verify the correctness of the migrated business attribute data.

[0156] Based on the above embodiments, the data migration module 340 can also be used for:

[0157] The data from the data migration out system is processed and transferred to the data migration in the migration script through the migration rule module. After obtaining the first-level migration result, the data migration in the migration in the migration script is processed and transferred to the data migration in the migration in the migration script through the verification rule module.

[0158] If the first-level migration result meets the verification criteria of the verification rules module, the migration script will transfer the data to the data migration system, and the data migration system will obtain the second-level migration result.

[0159] If the first-level migration result does not meet the verification criteria of the verification rules module, the migration script outputs an error result set and performs error processing on the error result set.

[0160] The data migration apparatus provided in the embodiments of the present invention can execute the data migration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0161] Example 4

[0162] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0163] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0164] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0165] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data migration methods.

[0166] Specifically, the method includes:

[0167] Based on the requirement elements of the data migration system under multiple requirement types, a hierarchical node diagram is constructed. The requirement types include data source, entity, and business attribute; different requirement types are located at different levels.

[0168] Obtain a standard data model that matches at least one data migration system. The standard data model is used to store the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different requirement types.

[0169] Based on the standard data models, a mapping relationship between each requirement element in adjacent levels is established in the hierarchical node graph to form a migration mapping relationship graph.

[0170] According to the migration mapping diagram, the data from each of the data migration-out systems is migrated to the data migration-in system.

[0171] In some embodiments, the data migration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data migration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data migration method by any other suitable means (e.g., by means of firmware).

[0172] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0177] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0178] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data migration method, characterized in that, include: Based on the requirement elements of the data migration system under multiple requirement types, construct a hierarchical node diagram. The requirement types include data source, entity, and business attribute. Different types of needs reside at different levels; Obtain a standard data model that matches at least one data migration system. The standard data model is used to store the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different requirement types. Based on the standard data models, a mapping relationship between each requirement element in adjacent levels is established in the hierarchical node graph to form a migration mapping relationship graph. According to the migration mapping diagram, the data from each of the data migration-out systems is migrated to the data migration-in system; According to the migration mapping diagram, the data from each of the data migration-out systems is migrated to the data migration-in system, including: A migration script is generated based on the migration mapping graph; wherein, the migration script includes a migration rule module and a verification rule module; The data from the data migration out system is processed by the migration rule module in the migration script and transferred to the data migration in the data migration in the data migration system, and the data migration in the data migration in the data migration system obtains the first-level migration result. The verification rules module is used to verify the correctness of migrated business attribute data.

2. The method according to claim 1, characterized in that, Based on the standard data model, establish the mapping relationship between each demand element in adjacent levels in the hierarchical node graph, including: Obtain the current standard data model corresponding to the data migration system currently being processed; In the order from low to high level, obtain the current processing level in the hierarchical node diagram, and obtain the target requirement type corresponding to the previous level adjacent to the current processing level. Based on the requirements elements and target requirement types in the current processing level, a search is performed in the current standard data model, and based on the search results, a mapping relationship is established between each requirement element in the current processing level and the matching requirement element in the previous level in the hierarchical node graph. The process returns to the previous one, proceeding from the lowest to the highest level, retrieving the operation at the current processing level in the hierarchical node graph, until processing reaches the second level of the hierarchical node graph.

3. The method according to claim 2, characterized in that, Based on the requirements elements and target requirement types in the current processing level, a search is performed in the current standard data model. Based on the search results, a mapping relationship is established in the hierarchical node graph between each requirement element in the current processing level and the matching requirement element in the previous level, including: Obtain the current requirement element in the current processing level, and obtain at least one element mapping relationship in the current standard data model that corresponds to both the current requirement element and the target requirement type; Based on the mapping relationship between the at least one element, determine the target upper-level requirement element that matches the current requirement element; Establish a mapping relationship between the current requirement element and the requirement element at the next higher level in the hierarchical node diagram; Return to the operation that retrieves the current required element in the current processing level, until all required elements in the current processing level have been processed.

4. The method according to claim 3, characterized in that, Retrieve the current requirement element at the current processing level, and obtain at least one element-to-element mapping relationship in the current standard data model that corresponds to both the current requirement element and the target requirement type, including: If the current requirement element has a mapping relationship with a unique target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matching target requirement type; If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry the same data, then the initial target requirement type is selected according to the data lineage, and the mapping relationship between the target requirement type and the current requirement element is used as the mapping relationship between the current requirement element and the matched target requirement type. If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types have a data migration relationship and carry different data, then the target requirement type that meets the rules is selected according to the preset rules, and the mapping relationship between the target requirement type and the current requirement element is taken as the mapping relationship between the current requirement element and the matched target requirement type.

5. The method according to claim 3, characterized in that, Retrieve the current requirement element at the current processing level, and obtain at least one element-to-element mapping relationship in the current standard data model that corresponds to both the current requirement element and the target requirement type, including: If the current requirement element has a mapping relationship with a unique target requirement type, then the mapping relationship is the mapping relationship between the current requirement element and the matching target requirement type; If the current requirement element has a mapping relationship with multiple target requirement types, and the multiple target requirement types do not have a data migration relationship, then all target requirement types are merged into the final target requirement type, and the mapping relationship between the target requirement type and the current requirement element is taken as the mapping relationship between the current requirement element and the matched target requirement type.

6. The method according to claim 1, characterized in that, The data from the data migration out system is processed and transferred to the data migration in the migration script through the migration rule module. After the data migration in the migration in the system obtains the first-level migration result, the process further includes: The first-level migration result is input into the migration script, and the verification rule module in the migration script processes and transmits it to the data migration system. If the first-level migration result meets the verification criteria of the verification rules module, the migration script will transfer the data to the data migration system, and the data migration system will obtain the second-level migration result. If the first-level migration result does not meet the verification criteria of the verification rules module, the migration script outputs an error result set and performs error processing on the error result set.

7. A data migration device, characterized in that, include: The hierarchical construction module is used to build a hierarchical node diagram based on the requirement elements of the data migration system under multiple requirement types. The requirement types include data sources, entities, and business attributes; different requirement types are located at different levels. The standard data model matching module is used to obtain the standard data model that matches at least one data migration system. The standard data model stores the mapping relationship between multiple elements of the data migration system. The two elements in each mapping relationship belong to different demand types. The migration mapping graph construction module is used to establish the mapping relationship between each requirement element in the adjacent levels of the hierarchical node graph according to each standard data model, thus forming a migration mapping graph. The data migration module migrates data from each of the data migration-out systems to the data migration-in system according to the migration mapping diagram. The data migration module is specifically used for: generating migration scripts based on the migration mapping graph; wherein, the migration scripts include a migration rule module and a verification rule module; The data from the data migration out system is processed by the migration rule module in the migration script and transferred to the data migration in the data migration in the data migration system, and the data migration in the data migration in the data migration system obtains the first-level migration result. The verification rules module is used to verify the correctness of migrated business attribute data.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the data migration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data migration method according to any one of claims 1-6.

Images