Data transplantation method, system and device and storage medium

Abstract

The invention discloses a data transplantation method, system and device and a storage medium. The method comprises the steps that a data transplantation request generated by a second system is sent to a first system; acquiring to-be-transplanted data acquired by the first system in response to the data transplantation request; querying a matched data transplantation template according to the transplantation scene; mapping the to-be-transplanted data to a second system according to a first data mapping rule preset in the data transplantation template and an access address of the second system; synchronously acquiring residual to-be-transplanted data, inputting the residual to-be-transplanted data into the dynamic mapping model, acquiring a second mapping rule, and mapping the residual to-be-transplanted data to a second system by utilizing the second mapping rule and an access address of the second system; and carrying out mapping result verification by utilizing the transplanted data mapped by the data verification standard in the data transplantation template, adjusting the first data mapping rule according to a verification result, and optimizing and training the dynamic mapping model. According to the method, data transplantation with high accuracy and high fitness can be completed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, device and storage medium for data migration. Background Art

[0002] Data migration technology plays a vital role in the modern information society, especially in the era of big data, the demand for data migration is growing. Whether it is the upgrading of internal enterprise systems or data sharing across platforms, efficient and accurate data migration has become one of the key factors to improve work efficiency. However, traditional data migration methods often rely on manual operations, which are not only time-consuming and labor-intensive, but also prone to errors, and cannot meet the needs of large-scale and frequent data migration.

[0003] Among existing data migration technologies, common practices include but are not limited to: manual export and import, use of general ETL tools (Extract, Transform, Load), and writing customized scripts to implement specific data migration tasks. Manual export and import is the most direct method, but it is inefficient and prone to errors; although ETL tools are powerful, they are still not flexible enough when dealing with complex data structures; and writing customized scripts requires high programming skills and has high maintenance costs.

[0004] Although the above methods can meet the basic needs of data migration to a certain extent, there are still many shortcomings. Especially in the face of changing data environments and complex business logic, these traditional methods often cannot guarantee the consistency and integrity of data, resulting in a large number of errors in the data migration process, which seriously affects the quality and availability of data. In addition, for the real-time migration of large-scale data, traditional methods are also difficult to achieve efficient and stable performance requirements. Therefore, how to improve the automation level and accuracy of data migration has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to achieve high-accuracy and strong adaptability data migration in the face of changing data environments and complex business logic, the present application provides a data migration method, system, device and storage medium.

[0006] In a first aspect, the present application provides a method for data migration, comprising: Sending a data migration request generated by the second system to the first system, the data migration request including a migration scenario, a data file name in the first system storing the migration data, a data file name in the second system requesting the migration data, and an access address of the second system; Acquire the data to be migrated obtained by the first system in response to the data migration request; query a matching data migration template from a data migration template library according to the migration scenario; the data migration template library stores data migration templates matching each migration scenario, and each data migration template includes a preset first data mapping rule and a data verification standard; According to the first data mapping rule preset in the queried data migration template and the access address of the second system, the data to be migrated is mapped to the second system; in the mapping process, part of the data to be migrated that cannot be mapped according to the preset first data mapping rule is synchronously obtained, recorded as the remaining data to be migrated, and the remaining data to be migrated is input into the dynamic mapping model constructed by machine learning technology to obtain the second mapping rule of the part of the data to be migrated, and the remaining data to be migrated is mapped to the second system by using the second mapping rule and the access address of the second system; The data verification standard in the data migration template is used to verify the mapping results of the migration data obtained in real time. If the mapping error rate is greater than the preset threshold period during the verification process, the mapping process of the data to be migrated is suspended. The first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be migrated, and the dynamic mapping model is optimized and trained. Then, the data to be migrated with a mapping error rate greater than the preset threshold period is re-mapped.

[0007] By adopting the above scheme, a hybrid mapping strategy is designed, which not only utilizes the preset configuration templates provided by different migration scenarios to quickly and accurately handle common data mapping tasks in corresponding scenarios, but also combines the dynamic mapping rule engine to complete the mapping rule acquisition of special fields or complex data structures to complete the mapping task, thereby improving the accuracy of data migration; and performs quality monitoring on data migration. Once the monitoring mapping performance deteriorates, the preset rules are immediately adjusted or the dynamic mapping engine is optimized to ensure the accuracy and efficiency of data migration.

[0008] Preferably, the method further includes: before mapping the data to be migrated to the second system, dividing the data to be migrated into migration tasks in advance, and mapping the data to be migrated to the second system according to the divided migration tasks; The data to be migrated is divided into migration tasks, including: inputting the data to be migrated into a migration data requirement acquisition model that matches the corresponding migration scenario, acquiring the requirement types of all the data to be migrated, and dividing some of the data to be migrated that belong to the same requirement type into the same migration task; the migration data classification model that matches each migration scenario adopts a neural network model, which is generated by training the migration data under the conditions of historical migration scenarios marked with high reliability requirements or high real-time requirements; The method of mapping the data to be migrated to the second system according to the migration task division includes: adaptively matching the second system interface supporting different transmission protocols according to the requirement type of the data to be migrated in each task, and each requirement type of the data to be migrated is matched with a second system interface supporting a class of transmission protocols; mapping the data to be migrated to the second system through the matching second system interface according to the first data mapping rule and the access address of the second system preset in the queried data migration template.

[0009] By adopting the above solution, the data to be migrated with different needs are reasonably divided and adaptively matched to the second system interface that supports different transmission protocols, ensuring that the advantages of various transmission protocols can be fully utilized during the data transmission process, meeting the needs of the data to be migrated, and improving the data transmission speed and stability.

[0010] Preferably, for different transplantation tasks, the types of data to be transplanted in different transplantation tasks are further divided into several transplantation subtasks; the types of data to be transplanted in different transplantation tasks are pre-set according to corresponding transplantation scenarios; The data type to be transplanted in each transplantation subtask is input into the transplantation data importance degree acquisition model matched with the corresponding transplantation scenario to obtain the importance degree of the data to be transplanted in each transplantation subtask; the transplantation data importance degree acquisition model matched with each transplantation scenario adopts a neural network model, which is generated by training the transplantation data type under the historical transplantation scenario conditions with the importance degree marked; The plurality of transplantation subtasks are sorted in descending order of importance, so that the transplantation subtasks with higher importance are executed with higher priority.

[0011] By adopting the above scheme, different migration tasks are further refined according to the migration data of different data types, and the importance judgment and sorting of the migration subtasks are completed, so that the migration subtasks with higher importance are executed first, thereby improving the efficiency and resource utilization of the entire data migration process.

[0012] Preferably, it also includes: Determine whether the amount of data to be transplanted in each transplantation subtask is greater than a preset threshold. If so, evenly divide the data to be transplanted in the transplantation subtask into several blocks, and transmit the mapped transplantation data in parallel on multiple transmission channels, each transmission channel including different routing nodes.

[0013] By adopting the above scheme, further division operations are automatically triggered according to the amount of data to be transplanted in the transplantation subtask, so as to avoid excessive occupation of corresponding transplantation data resources after mapping due to excessive data volume in a single task, and the transplantation data transmission that completes the mapping is carried out in parallel on multiple transmission channels to improve the data transmission speed.

[0014] Preferably, it also includes: The transmission time of the transplanted data mapped corresponding to each partition block is counted and cluster analysis is performed to determine whether there is a partition block whose corresponding transmission time is greater than the preset threshold from the cluster center. If so, other transmission channels are selected to replace the original transmission channel of the current partition block to complete the transmission of the partition block.

[0015] By adopting the above scheme, the partition blocks with abnormal transmission duration are discovered in time to trigger the selection of other transmission channels to replace the current transmission channel, thereby avoiding data transmission failure caused by failure of a single transmission channel and ensuring smooth data migration.

[0016] Preferably, it also includes: In the process of mapping the data to be migrated to the second system according to the migration task, the status of the data to be migrated corresponding to different migration tasks is displayed, and the status includes: a migration completed status, a migration pending status, and a migration in progress status.

[0017] By adopting the above solution, the status of the data to be migrated corresponding to different migration tasks is displayed, so that the user can clearly understand the status of each migration task, thereby improving the transparency and controllability of data migration.

[0018] Preferably, a data migration log is generated according to the mapping result of the data to be migrated, and the data migration log includes: migration time, successfully migrated data and migration success rate.

[0019] By adopting the above solution, a data migration log is generated to comprehensively monitor and record the data migration process, which is convenient for subsequent data review and problem troubleshooting, and effectively improves the transparency and credibility of data migration.

[0020] In a second aspect, the present application provides a data migration system, comprising: A data migration request sending module, used to send a data migration request generated by the second system to the first system, wherein the data migration request includes a migration scenario, a data file name in the first system storing the migration data, a data file name in the second system requesting the migration data, and an access address of the second system; A module for acquiring data to be migrated, used for acquiring the data to be migrated acquired by the first system in response to the data migration request; The data to be transplanted mapping module is used to query the matching data transplantation template from the data transplantation template library according to the transplantation scenario; the data transplantation template library stores the data transplantation template matching each transplantation scenario, and each data transplantation template includes a preset first data mapping rule and a data verification standard; the data to be transplanted is mapped to the second system according to the preset first data mapping rule in the queried data transplantation template and the access address of the second system; the part of the data to be transplanted that cannot be mapped according to the preset first data mapping rule during the mapping process is synchronously obtained, recorded as the remaining data to be transplanted, the remaining data to be transplanted is input into the dynamic mapping model constructed by using the machine learning technology, the second mapping rule of the part of the data to be transplanted is obtained, and the remaining data to be transplanted is mapped to the second system by using the second mapping rule and the access address of the second system; The transplantation data verification module is used to verify the mapping results of the transplantation data that has been mapped in real time using the data verification standards in the data transplantation template. If the mapping error rate is greater than a preset threshold period during the verification process, the mapping process of the data to be transplanted is suspended, and the first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be transplanted, and the dynamic mapping model is optimized and trained, and then the data to be transplanted with a mapping error rate greater than the preset threshold period is re-mapped.

[0021] By adopting the above solution, using the data migration template and dynamic mapping model in the data mapping module to be migrated, data migration in various migration scenarios can be achieved, and the consistency and integrity of the migrated data can be ensured, thereby achieving efficient and accurate data migration.

[0022] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0023] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0024] In summary, this application has the following beneficial effects: 1. Design a hybrid mapping strategy, using data migration templates and dynamic mapping models, which can flexibly cope with various migration scenarios, ensure data consistency and integrity, solve the shortcomings of traditional methods in dealing with complex data structures, and improve data migration accuracy; through the data verification mechanism, real-time monitoring and correction of errors in the data mapping process, timely adjustment of preset rules or optimization of dynamic mapping engines, improve the stability and reliability of data migration; 2. Divide the tasks of the data to be migrated according to the transmission requirements of the data to be migrated and the importance of different types of data to be migrated, adaptively match the second system interface that supports different transmission protocols, complete the task execution sequence determination, and improve the data transmission speed and stability; 3. During the data migration process, the status of the data to be migrated corresponding to different migration tasks is displayed, so that users can clearly understand the status of each migration task; and by generating a detailed migration data log for the migration results, it is convenient for subsequent data review and troubleshooting, helping users to control the data migration process and results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the data migration method described in a specific embodiment; Figure 2 is a flow chart of data mapping in the data migration method described in a specific embodiment; Figure 3 It is a structural schematic diagram of the data migration system described in a specific embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, the embodiment of the present application discloses a method for data migration, and the specific steps include: S1. Send a data migration request.

[0028] Specifically, data migration generally refers to the process of moving data between different storage locations, formats or systems. In this embodiment, taking data migration between different systems as an example, a first system is used as a data source system, and a second system is used as a migration target system. A data migration interface is designed in the second system, so that the user can input through the migration interface: a migration scenario, a file name (or table name) in the second system where the migration data requested is requested, a file name (or table name) in the first system where the migration data requested is stored, and an access address of the second system where the migration data requested is requested, thereby generating a data migration request.

[0029] Among them, the migration scenario adopts common migration scenario types, mainly including: in the system upgrade and replacement scenario, the data in the old system is migrated to the new system, and the main transmission content includes key business data such as user information, transaction records, configuration settings, etc.; in the database system migration scenario, the data of one database system is migrated to another database system, and the main transmission content includes the table structure, data records, stored procedures, etc. in the database; in the cloud platform system migration scenario, the data is migrated from the application platform system of one provider to the application platform system of another provider, and the main transmission content includes: user information, transaction data, file storage, etc.

[0030] By accessing the second system, the data migration request generated by the second system is sent to the first system, so that the first system responds to the corresponding data migration request, thereby querying the corresponding requested migration data according to the file name (or table name) in the first system storing the requested migration data in the data migration request, and recording it as the data to be migrated.

[0031] S2. Obtain the data migration request response result.

[0032] Specifically, after responding to the data migration request generated by the second system, the first system queries the migration data corresponding to the request as a response result, and obtains the data to be migrated obtained by the first system in response to the data migration request.

[0033] S3. Complete data mapping according to the data migration request and the data migration request response result.

[0034] In order to improve the data mapping rate, a data migration template library can be built based on typical and common data mapping rules in different migration scenarios, and data mapping can be quickly completed according to the preset mapping rules in the template, such as Figure 2 As shown, the specific steps include: S31. Query a matching data migration template from a data migration template library according to the migration scenario.

[0035] Specifically, the data migration template library stores data migration templates matching each migration scenario, and each data migration template includes a preset first data mapping rule and a data verification standard adapted to the current migration scenario. Each data migration template uses a configurable template, and when a data migration template corresponding to a migration scenario is queried, it can be directly displayed on the data migration interface of the second system, so that users can add custom mapping rules to the queried data migration template according to their own needs.

[0036] For example: the first preset mapping rules preset in the first data migration template matching the database system migration scenario include: field correspondence rules, data type and storage format conversion rules, etc., and the first data verification standard includes integrity, accuracy and consistency verification, etc.; the first preset mapping rules preset in the second data migration template matching the cloud platform system migration scenario include: access permission mapping rules, field correspondence rules, data type and storage format conversion rules, etc., and the second data verification standard includes availability verification, integrity, accuracy and consistency verification, etc.; the first mapping rules preset in the third data migration template matching the system upgrade and replacement scenario include: entity relationship mapping rules, field correspondence rules, data type and storage format conversion rules, etc., and the third data verification standard includes: business logic verification, integrity, accuracy and consistency verification, etc.

[0037] S32: Map the data to be migrated to the second system according to the first data mapping rule preset in the queried data migration template and the access address of the second system.

[0038] Specifically, the data to be transplanted is mapped and converted according to the queried preset first data mapping rule, and the data after mapping conversion is loaded and stored into the storage location corresponding to the file name in the second system requesting the transplanted data through the access address of the second system.

[0039] For example: for the data to be migrated (user information table) in the first database system, including fields such as user ID, user name, user email, etc., according to the field correspondence rules in the preset first data mapping rules, the data to be migrated and the data to be migrated (customer information table) in the second database system are mapped and data type converted, the user ID corresponds to the customer number, and the user name corresponds to the customer name.

[0040] S33, synchronously acquiring part of the data to be migrated that cannot be mapped according to the preset first data mapping rule during the mapping process.

[0041] Considering the existence of special fields or complex data structures (such as nested fields), a dynamic mapping engine is selected for mapping, thereby extracting part of the data to be migrated that cannot be mapped according to the first data mapping rule preset in the data migration template library in advance, and recording part of the data to be migrated as Remaining data to be migrated. For example, it is difficult to map the user mailbox in the first database system and the user contact number in the second database system according to the field correspondence rules, and the data is recorded as remaining data to be migrated.

[0042] S34: input the remaining data to be transplanted into a dynamic mapping model constructed by using machine learning technology to obtain a second mapping rule for the part of the data to be transplanted.

[0043] Specifically, a dynamic mapping model is constructed using machine learning technology, and a deep learning neural network algorithm is specifically used to construct the dynamic mapping model. The input of the dynamic mapping model is the data information in the file corresponding to the file name of the data to be transplanted and the system file name requesting the transplanted data, and the output is the mapping rule for the data to be transplanted, recorded as the second mapping rule; the model is mainly trained and generated through data to be transplanted, mapped data and corresponding manually set mapping rules for special fields or complex data structures in different transplantation scenarios.

[0044] The built and trained dynamic mapping model is used to obtain the second mapping rules for the remaining data to be migrated, such as generating field association rules for the user mailbox in the first database system and the user contact number in the second database system.

[0045] S35 , mapping the remaining data to be migrated to the second system using the second mapping rule and the access address of the second system.

[0046] Specifically, the remaining transplant data is mapped and converted according to the acquired second data mapping rule, and the data after the mapping conversion is loaded and stored into the storage location corresponding to the file name in the second system requesting the transplant data through the access address of the second system.

[0047] S4. Monitor the data mapping process in real time, verify the data mapping results and adjust the mapping rules.

[0048] During the migration process, in order to ensure the accuracy of data migration, the data mapping results are monitored in real time; if any anomalies or errors are found, the mapping rules and conversion strategies are adjusted in real time to ensure the correct migration of data. The specific steps include: The mapping result of the transplanted data obtained in real time is verified by using the data verification standard in the data transplantation template. During the verification process, if the mapping error rate is greater than the preset threshold period, an interval period is set to determine the mapping error rate of the transplanted data in the interval period. If the mapping error rate is greater than the preset threshold, it indicates that there are many mapping errors in the current period and the mapping performance is poor, so the mapping process of the transplanted data is suspended. For example, if the mapping error rate exceeds 5% in a certain time period, the mapping will be suspended, the mapping rules will be adjusted, and the dynamic mapping model will be retrained. The data to be transplanted in the corresponding time period will be re-mapped before the mapping is continued.

[0049] The first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be transplanted. Specifically, the mapping rules corresponding to the failed mapping and the successful mapping in the data to be transplanted can be analyzed, and the mapping rules with failed mapping can be deleted from the first data mapping rule, and the mapping priority of the mapping rule corresponding to the successful mapping of the data to be transplanted can be increased. In addition to adjusting the first data mapping rule, the mapping rule corresponding to the successful mapping of the data to be transplanted can be used as training data or additional training data to perform incremental optimization training on the dynamic mapping model. After adjustment, the data to be transplanted whose mapping error rate is greater than a preset threshold period is remapped, and the remapped data replaces the original mapping data and is loaded and stored in the storage location corresponding to the file name in the second system requesting the transplanted data.

[0050] The corresponding monitoring data mapping process ends when all the data to be migrated are mapped.

[0051] S5. Generate a data migration log.

[0052] Specifically, in order to facilitate the user's subsequent data review and troubleshooting, after all the data to be migrated are mapped and the corresponding monitoring data mapping process is ended, the mapping results of the data to be migrated are statistically analyzed and a data migration log is generated. The data migration log includes: migration time, successfully migrated data and migration success rate.

[0053] In a specific embodiment, in order to further improve the efficiency of data migration and meet the needs of users for data migration, the function of dividing the data migration tasks and matching the data transmission interface as needed can be added, and the method further includes: Before mapping the data to be migrated to the second system, the data to be migrated is divided into migration tasks in advance, and the data to be migrated is mapped to the second system according to the divided migration tasks, and the corresponding tasks with different requirements of the data to be migrated are respectively selected to accelerate the mapping of the data to be migrated to the second system.

[0054] The step of dividing the data to be migrated into migration tasks includes: The data to be transplanted are input into the transplantation data demand acquisition model that matches the corresponding transplantation scenario to obtain the demand types of all the data to be transplanted; the demand types of all the data to be transplanted include: high reliability demand or high real-time demand; the transplantation data classification model that matches each transplantation scenario adopts a neural network model, which is generated by training the transplantation data under the conditions of historical transplantation scenarios marked with high reliability demand or high real-time demand; some of the data to be transplanted that belong to the same demand type are divided into the same transplantation task, such as: the first transplantation task corresponding to the data to be transplanted with high reliability demand type, and the second transplantation task corresponding to the data to be transplanted with protective gear with high real-time demand.

[0055] Among them, mapping the data to be migrated to the second system according to the migration task division includes: adaptively matching the second system interface supporting different transmission protocols according to the demand type of the data to be migrated in each task, and each demand type of the data to be migrated is matched with a second system interface supporting a class of transmission protocols, such as: an interface supporting the TCP protocol that matches the high reliability demand type, and an interface supporting the UDP protocol that matches the high real-time demand type.

[0056] According to the preset first data mapping rules in the queried data migration template and the access address of the second system, the data to be migrated is mapped to the second system through the matching second system interface. For example, for the mapping of the data to be migrated to the second system in the first migration task, the data in the format of the TCP transmission protocol after the mapping conversion can be loaded and stored into the storage location corresponding to the file name in the second system requesting the data to be migrated through the matching interface supporting the TCP protocol, thereby ensuring the reliable transmission of the data; for the mapping of the data to be migrated to the second system in the second migration task, the data in the format of the UDP transmission protocol after the mapping conversion can be loaded and stored into the storage location corresponding to the file name in the second system requesting the data to be migrated through the matching interface supporting the UDP protocol, thereby ensuring the high real-time migration of the data and improving the efficiency of data migration.

[0057] In addition, in the process of mapping the data to be migrated to the second system according to the migration task division, the status of the data to be migrated corresponding to different migration tasks is displayed. Specifically, it can be displayed in the data migration interface of the second system. The progress of each migration task can be displayed using a progress bar. The progress bar in different states has different display colors. The states include: completed migration state, pending migration state, and ongoing migration state, so that users can view the progress of the current data migration in real time.

[0058] In a specific embodiment, in order to further achieve orderly and efficient data migration, the functions of data sorting and parallel transmission are selected to be added, and the method further includes: For different migration tasks, the data types to be migrated in different migration tasks are further divided into several migration subtasks. Among them, the data types to be migrated in different migration tasks are pre-set according to the corresponding migration scenarios, that is, according to the typical data types to be migrated in different migration scenarios, the data types to be migrated in the current migration task are matched and judged, such as: the typical data types to be migrated in the database system migration scenario include: structured types such as table structure, index, data record, historical transaction data type, user information data type, log and audit type, etc.; the typical data types to be migrated in the system upgrade and replacement scenario include: configuration file type, log file type, business data and transaction type, etc.; the common data types to be migrated in the cloud platform system migration scenario include: identity authentication and access control data type, database and storage data type, network and configuration data type, etc.

[0059] Alternatively, the types of data to be transplanted in the different transplantation tasks are obtained through a transplantation data sub-classification model that matches the corresponding transplantation scenario, and the data to be transplanted in the different transplantation tasks are respectively input into the transplantation data classification model that matches the corresponding transplantation scenario to obtain the types of data to be transplanted in the different transplantation tasks; the transplantation data sub-classification model that matches each transplantation scenario is generated by training the data to be transplanted that are annotated with common transplantation data types under historical transplantation scenarios.

[0060] The data type to be transplanted in each transplantation subtask is input into the transplantation data importance degree acquisition model matched with the corresponding transplantation scenario, and the importance degree of the data to be transplanted in each transplantation subtask is obtained, including: very important, generally important, unimportant, etc.; the transplantation data importance degree acquisition model matched with each transplantation scenario adopts a neural network model, which is generated by training the transplantation data type under the historical transplantation scenario conditions with the importance degree marked; Several transplantation subtasks are sorted in descending order of importance. Tasks with the same importance can be sorted according to the order of storage time of the transplantation data or randomly sorted, so that the transplantation subtask with higher importance will be executed with higher priority, that is, the data to be transplanted in the transplantation subtask will be mapped to the second system for processing.

[0061] In addition, in addition to giving priority to the data to be migrated corresponding to each migration subtask with high importance, The method further comprises: It is determined whether the amount of data to be transplanted in each transplantation subtask is greater than a preset threshold. If so, it indicates that the single task has too much data. To avoid excessive occupation of the corresponding mapped transplantation data resources, the data to be transplanted in the transplantation subtask is evenly divided into several blocks, and the mapped transplantation data is transmitted in parallel on multiple transmission channels and mapped to the second system in time sequence. Each transmission channel includes different routing nodes to disperse the network load and reduce the risk of single point failure, thereby ensuring the reliability and efficiency of the entire data transplantation process.

[0062] In order to further avoid data transmission failure caused by transmission channel failure and ensure smooth data migration, the method further includes: The transmission time of the transplanted data mapped to each partition block is counted and cluster analysis is performed, wherein the transmission time includes: the time from the start of the transplanted data corresponding to the data to be transplanted in each partition block to the transmission mapping to the second system; determining whether there is a partition block whose corresponding transmission time is greater than a preset threshold from the cluster center, then selecting other transmission channels to replace the original transmission channel of the current partition block to complete the transmission of the partition block.

[0063] like Figure 3 As shown, the embodiment of the present application discloses a data migration system, which is connected to the first subsystem and the second subsystem in communication, as middleware, and specifically includes: A data migration request sending module 101 is used to send a data migration request generated by the second system to the first system, wherein the data migration request includes a migration scenario, a data file name in the first system storing the migration data, a data file name in the second system requesting the migration data, and an access address of the second system; The to-be-migrated data acquisition module 102 is used to acquire the to-be-migrated data acquired by the first system in response to the data migration request; The data mapping module 103 for transplantation is used to query the matching data transplantation template from the data transplantation template library according to the transplantation scenario; the data transplantation template library stores the data transplantation template matching each transplantation scenario, and each data transplantation template includes a preset first data mapping rule and a data verification standard; the data to be transplanted is mapped to the second system according to the preset first data mapping rule in the queried data transplantation template and the access address of the second system; the part of the data to be transplanted that cannot be mapped according to the preset first data mapping rule during the mapping process is synchronously obtained, recorded as the remaining data to be transplanted, the remaining data to be transplanted is input into the dynamic mapping model constructed by using the machine learning technology, the second mapping rule of the part of the data to be transplanted is obtained, and the remaining data to be transplanted is mapped to the second system by using the second mapping rule and the access address of the second system; The transplantation data verification module 104 is used to verify the mapping results of the transplantation data that has been mapped and acquired in real time by using the data verification standard in the data transplantation template. If the mapping error rate is greater than a preset threshold period during the verification process, the mapping process of the data to be transplanted is suspended, and the first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be transplanted, and the dynamic mapping model is optimized and trained, and then the data to be transplanted with a mapping error rate greater than the preset threshold period is re-mapped.

[0064] The system further comprises: The data migration log generating module 105 is used to generate a data migration log according to the mapping result of the data to be migrated. The data migration log includes: migration time, successfully migrated data and migration success rate.

[0065] The data to be transplanted partitioning module 106 is used to divide the data to be transplanted into transplantation tasks before mapping the data to be transplanted to the second system, and map the data to be transplanted to the second system according to the divided transplantation tasks; the transplantation task partitioning of the data to be transplanted includes: inputting the data to be transplanted into the transplantation data demand acquisition model matched with the corresponding transplantation scenario, obtaining the demand type of all the data to be transplanted, and dividing part of the data to be transplanted belonging to the same type of demand into the same transplantation task; the transplantation data classification model matched with each transplantation scenario adopts a neural network model, which is generated by training transplantation data under historical transplantation scenario conditions marked with high reliability requirements or high real-time requirements; the mapping of the data to be transplanted to the second system according to the transplantation task partitioning includes: adaptively matching the second system interface supporting different transmission protocols according to the demand type of the data to be transplanted in each task, and matching each demand type of the data to be transplanted with a second system interface supporting a type of transmission protocol; according to the first data mapping rule preset in the queried data transplantation template and the access address of the second system, mapping the data to be transplanted to the second system through the matched second system interface.

[0066] In a specific embodiment, the data to be transplanted division module 106 is also used to further divide the data to be transplanted in different transplantation tasks into a number of transplantation subtasks according to the type of data to be transplanted in different transplantation tasks; the type of data to be transplanted in different transplantation tasks is pre-set according to the corresponding transplantation scenario; the type of data to be transplanted in each transplantation subtask is input into the transplantation data importance degree acquisition model matched with the corresponding transplantation scenario to obtain the importance degree of the data to be transplanted in each transplantation subtask; the transplantation data importance degree acquisition model matched with each transplantation scenario adopts a neural network model, which is generated by training the transplantation data type under the historical transplantation scenario conditions with the importance degree marked; and the number of transplantation subtasks are sorted in order from large to small in importance, so that the transplantation subtask with higher importance degree is executed with higher priority.

[0067] The transplant data parallel transmission module 107 is used to determine whether the amount of data to be transplanted in each transplant subtask is greater than a preset threshold value. If so, the data to be transplanted in the transplant subtask is evenly divided into several blocks, and the transplant data mapped is transmitted in parallel on multiple transmission channels, each transmission channel including different routing nodes; it is also used to count the transmission time of the transplant data mapped corresponding to each divided block and perform cluster analysis to determine whether there is a divided block whose corresponding transmission time is greater than the distance from the cluster center by a preset threshold value, and then select other transmission channels to replace the original transmission channel of the current divided block to complete the transmission of the divided block.

[0068] The embodiment of the present application also discloses a computer-readable storage medium.

[0069] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute a method such as the above-mentioned data migration. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0070] The embodiment of the present application also discloses a computer device.

[0071] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned data migration method.

[0072] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A method for data migration, characterized in that: include: Sending a data migration request generated by the second system to the first system, the data migration request including a migration scenario, a data file name in the first system storing the migration data, a data file name in the second system requesting the migration data, and an access address of the second system; Acquire the data to be migrated obtained by the first system in response to the data migration request; query a matching data migration template from a data migration template library according to the migration scenario; the data migration template library stores data migration templates matching each migration scenario, and each data migration template includes a preset first data mapping rule and a data verification standard; Mapping the data to be migrated to the second system according to the first data mapping rule preset in the queried data migration template and the access address of the second system; During the synchronous acquisition mapping process, part of the data to be migrated that cannot be mapped according to the preset first data mapping rule is recorded as the remaining data to be migrated, the remaining data to be migrated is input into the dynamic mapping model constructed by machine learning technology, the second mapping rule of the part of the data to be migrated is obtained, and the remaining data to be migrated is mapped to the second system by using the second mapping rule and the access address of the second system; The data verification standard in the data migration template is used to verify the mapping results of the migration data obtained in real time. If the mapping error rate is greater than the preset threshold period during the verification process, the mapping process of the data to be migrated is suspended. The first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be migrated, and the dynamic mapping model is optimized and trained. Then, the data to be migrated with a mapping error rate greater than the preset threshold period is re-mapped.

2. The data migration method according to claim 1, characterized in that: Also includes: Before mapping the data to be migrated to the second system, the data to be migrated is divided into migration tasks in advance, and the data to be migrated is mapped to the second system according to the divided migration tasks; The data to be migrated is divided into migration tasks, including: inputting the data to be migrated into a migration data requirement acquisition model that matches the corresponding migration scenario, acquiring the requirement types of all the data to be migrated, and dividing some of the data to be migrated that belong to the same requirement type into the same migration task; the migration data classification model that matches each migration scenario adopts a neural network model, which is generated by training the migration data under the conditions of historical migration scenarios marked with high reliability requirements or high real-time requirements; The method of mapping the data to be migrated to the second system according to the migration task division includes: adaptively matching the second system interface supporting different transmission protocols according to the requirement type of the data to be migrated in each task, and each requirement type of the data to be migrated is matched with a second system interface supporting a class of transmission protocols; mapping the data to be migrated to the second system through the matching second system interface according to the first data mapping rule and the access address of the second system preset in the queried data migration template.

3. The data migration method according to claim 2, characterized in that: Also includes: For different transplantation tasks, the types of data to be transplanted in different transplantation tasks are further divided into several transplantation subtasks; the types of data to be transplanted in different transplantation tasks are pre-set according to corresponding transplantation scenarios; The data type to be transplanted in each transplantation subtask is input into the transplantation data importance degree acquisition model matched with the corresponding transplantation scenario to obtain the importance degree of the data to be transplanted in each transplantation subtask; the transplantation data importance degree acquisition model matched with each transplantation scenario adopts a neural network model, which is generated by training the transplantation data type under the historical transplantation scenario conditions with the importance degree marked; The plurality of transplantation subtasks are sorted in descending order of importance, so that the transplantation subtasks with higher importance are executed with higher priority.

4. The method for data migration according to claim 3, characterized in that: Also includes: Determine whether the amount of data to be transplanted in each transplantation subtask is greater than a preset threshold. If so, evenly divide the data to be transplanted in the transplantation subtask into several blocks, and transmit the mapped transplantation data in parallel on multiple transmission channels, each transmission channel including different routing nodes.

5. The method for data migration according to claim 4, characterized in that: Also includes: The transmission time of the transplanted data mapped corresponding to each partition block is counted and cluster analysis is performed to determine whether there is a partition block whose corresponding transmission time is greater than the preset threshold from the cluster center. If so, other transmission channels are selected to replace the original transmission channel of the current partition block to complete the transmission of the partition block.

6. The data migration method according to claim 2, characterized in that: Also includes: In the process of mapping the data to be migrated to the second system according to the migration task, the status of the data to be migrated corresponding to different migration tasks is displayed, and the status includes: a migration completed status, a migration pending status, and a migration in progress status.

7. The data migration method according to claim 1, characterized in that: Also includes: A data migration log is generated according to the mapping result of the data to be migrated, and the data migration log includes: migration time, successfully migrated data and migration success rate.

8. A data migration system, characterized in that: include: A data migration request sending module, used to send a data migration request generated by the second system to the first system, wherein the data migration request includes a migration scenario, a data file name in the first system storing the migration data, a data file name in the second system requesting the migration data, and an access address of the second system; A module for acquiring data to be migrated, used for acquiring the data to be migrated acquired by the first system in response to the data migration request; The data to be migrated mapping module is used to query the matching data migration template from the data migration template library according to the migration scenario; the data migration template library stores the data migration template matching each migration scenario, and each data migration template includes a preset first data mapping rule and a data verification standard; the data to be migrated is mapped to the second system according to the preset first data mapping rule in the queried data migration template and the access address of the second system; During the synchronous acquisition mapping process, part of the data to be migrated that cannot be mapped according to the preset first data mapping rule is recorded as the remaining data to be migrated, the remaining data to be migrated is input into the dynamic mapping model constructed by machine learning technology, the second mapping rule of the part of the data to be migrated is obtained, and the remaining data to be migrated is mapped to the second system by using the second mapping rule and the access address of the second system; The transplantation data verification module is used to verify the mapping results of the transplantation data that has been mapped in real time using the data verification standards in the data transplantation template. If the mapping error rate is greater than a preset threshold period during the verification process, the mapping process of the data to be transplanted is suspended, and the first data mapping rule is adjusted according to the mapping rule corresponding to the successful mapping of the data to be transplanted, and the dynamic mapping model is optimized and trained, and then the data to be transplanted with a mapping error rate greater than the preset threshold period is re-mapped.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images