Data migration method, data migration system, electronic device and storage medium

Abstract

The embodiments of this specification disclose a data migration method, a data migration system, an electronic device, and a storage medium, which belong to the field of data processing technology and can realize a unified data migration method for multiple businesses, thereby improving data migration efficiency, security, and implementation costs. The method comprises: obtaining source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities; compiling the source task data into a mirror file stored by a source server, and generating a mirror compilation event for the task entity to be migrated corresponding to the source task data; based on the mirror compilation event, migrating the mirror file stored on the source server to a target server, and generating a mirror migration event for the task entity to be migrated, wherein the mirror migration event is used to indicate that the mirror file stored on the target server is restored to the target task environment.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a data migration method, a data migration system, an electronic device, and a storage medium. Background Art

[0002] With the accelerated development of next-generation information and communication technologies like big data and cloud computing, the pace of innovation and iteration in new information business applications is accelerating, while the scale and volume of data used in these applications are growing exponentially. To analyze business competitiveness information from large-scale data resources, gain insights into business service issues, and support decision-making, enterprises and institutions often use data warehouses to store data from business applications, relational databases, and external data sources, enabling data analysis and insights.

[0003] During the application innovation and update iteration process of enterprises and institutions, most innovative applications and iteratively updated applications need to be verified in a test environment. These verified innovative applications and iteratively updated applications are then deployed to the production environment to control security risks and reduce the impact on services in the production environment. Therefore, data migration between different environments is inevitable. Enterprises and institutions will face the need to migrate massive amounts of data and various computer files, including user data, interface mapping data, applications, script files, dependency libraries, databases, and data warehouses, which constitute innovative or iteratively updated applications. Currently, technicians often manually migrate the changed data item by item. However, the data scale of enterprises and institutions is huge, the mapping relationships are complex, and the data migration methods for different businesses are diverse. Manual item-by-item migration is not only cumbersome and inefficient, but also very prone to errors, causing security issues for services in the production environment. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a data migration method, a data migration system, an electronic device, and a storage medium to avoid migration errors caused by different migration methods of the data to be migrated, thereby realizing a unified data migration method for multiple businesses and improving data migration efficiency, security, and implementation costs.

[0005] In order to achieve the above objectives, the embodiments of this specification adopt the following scheme:

[0006] In a first aspect, a data migration method is provided, the data migration method comprising:

[0007] Acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, and the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment;

[0008] Compiling the source task data into an image file stored by a source server, and generating an image compilation event for a task entity to be migrated corresponding to the source task data;

[0009] Based on the image compilation event, the image file stored in the source server is migrated to the target server, and an image migration event of the task entity to be migrated is generated. The image migration event is used to instruct to restore the image file stored in the target server to the target task environment.

[0010] In a second aspect, a data migration system is provided, the data migration system comprising:

[0011] An acquisition module is configured to acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, wherein the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on the task classification of the source task environment;

[0012] An image compilation module, configured to compile the source task data into an image file stored by a source server, and generate an image compilation event for a task entity to be migrated corresponding to the source task data;

[0013] The image migration module is used to migrate the image file stored in the source server to the target server based on the image compilation event, and generate an image migration event for the task entity to be migrated. The image migration event is used to restore the image file stored in the target server to the target task environment.

[0014] According to a third aspect, an electronic device is provided, comprising:

[0015] at least one processor;

[0016] a memory connected to the at least one processor;

[0017] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the aforementioned method by executing the instructions stored in the memory.

[0018] According to a fourth aspect, a machine-readable storage medium is provided, which stores machine instructions. When the machine instructions are executed on a machine, the machine executes the aforementioned method.

[0019] In the scheme of the embodiments of this specification, in the task data stored in the source data warehouse, the data to be migrated in the source task environment is divided into source data of multiple task entities according to the task classification of the source task environment. The divided migrated data are aggregated according to the data range involved in the task classification through the task entity, and a mapping is established based on the task classification in the source task environment to keep the data under each business function aggregated into the corresponding task entity, providing a basis for avoiding data intersection between different task classifications. At the same time, the source data warehouse is used to store and obtain task data during migration, without the need to introduce an external or third-party data warehouse, and manual screening is not performed from the data to be migrated by manually comparing the differences such as data changes and file changes.

[0020] In the solution of the embodiment of this specification, after obtaining the source task data, the source task data corresponding to the task entity to be migrated is uniformly compiled into a mirror file, and a mirror compilation event is generated. Based on the task entities divided into different task categories, a unified migration transition environment can be provided through the mirror file. The source task data in the migration transition environment all correspond to the task entity to be migrated, avoiding data overlap between different task categories, and automatically migrating through the mirror compilation event instruction, rather than manually processing each item according to the difference data item, nor processing in batches.

[0021] In the solution of the embodiment of this specification, after the mirror compilation event is generated, the objects of data migration are the source server and the target server. At the same time, the data object to be migrated is the mirror file, which can unify the migration operations of various data, large-scale data, and complex mapping between data into the mirror file transfer operation between the source server and the target server, and does not need to perform different migrations according to the characteristics of data style, size, mapping method, etc. This avoids the errors caused by different migration methods and the confusion of business functions in the target task environment, so that after the migrated task data is restored in the target task environment, the same business functions as those in the source task environment can be provided, avoiding the interruption of service use in the target task environment. The generated events can reflect the stage characteristics of the migration and are conducive to determining the task entity and stage where the error occurs. By using the task entity as the basic granularity of the migration processing, there is no need to compare and search for data changes / file changes item by item, and a unified migration method between multiple businesses is realized, which has the characteristics of high migration efficiency and reduces the cost of data migration implementation for users, while providing long-term evolution support for the application innovation and update iteration of enterprises and institutions.

[0022] Other features and advantages of the embodiments of this specification will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0024] Figure 1 This is a schematic diagram of steps of an exemplary data migration method according to an embodiment of this specification;

[0025] Figure 2 This is a schematic diagram of an exemplary data graph structure of an embodiment of this specification;

[0026] Figure 3 This is a schematic diagram of an exemplary data graph structure of an embodiment of this specification;

[0027] Figure 4 This is a schematic diagram of an exemplary data graph structure of an embodiment of this specification;

[0028] Figure 5 This is a schematic diagram of an exemplary selection window interface according to an embodiment of this specification;

[0029] Figure 6 This is a module diagram of an exemplary data migration system according to an embodiment of this specification;

[0030] Figure 7 This is a module diagram of an exemplary data migration system according to an embodiment of this specification;

[0031] Figure 8 This is a module diagram of an exemplary data migration system according to an embodiment of this specification;

[0032] Figure 9 This is a module diagram of an exemplary data migration system according to an embodiment of this specification;

[0033] Figure 10 This is a module diagram of an exemplary data migration system according to an embodiment of this specification;

[0034] Figure 11 This is a schematic diagram of a module of an exemplary electronic device according to an embodiment of this specification. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0036] As mentioned above, enterprises and institutions need to migrate data between different environments, requiring consistency in data formats, data integrity, data / file mappings, and conversion rule files. This ensures that the migrated environment provides the same business functionality as the source environment. However, due to practical factors such as large data volumes, diverse data formats, and complex data sources, manual data migration by technical personnel is difficult and tedious.

[0037] In some possible examples of manual data migration, first, within the source environment, you can compare the differences across multiple database tables, field by field, or row by row, and then perform import and export operations using database commands to obtain the exported database data. Second, you can compare the business applications, data mapping files, external interface files, and data conversion rule files that use the database with the programs and files in the environment to be migrated, and manually transfer the changed files to the environment to be deployed. After migration, if the migrated data in the migrated environment does not have business functionality, you can also try to find and specify the link configuration in the migrated environment to establish the business functionality. However, completing this simple migration of just one business function requires migrating the remaining data that has changed, which is extremely time-consuming, labor-intensive, and costly. Furthermore, errors in any program, file, or other data can easily occur during the data migration process. If not discovered in time, it is difficult to pinpoint the stage and location of the error. To ensure the security and availability of the migrated environment, you will have to restore the pre-migration deployment environment, reset the configuration, and re-migrate the entire data, making the migration efficiency unacceptable to enterprises and institutions.

[0038] In addition, you can also try to use the alternative solution of database migration script processing. The database migration script can generate the target data table in the target database based on the data structure information of the source data table in the source database, and migrate multiple data files and multiple data structure files to the target database through import and export operations. However, first, enterprises and institutions use data warehouses for data storage, and the number and variety of databases used are large, such as relational databases and key-value databases. The database migration script is not universal. The migration pair between the source relational database and the target relational database requires a separate configuration script, and the migration pair between the source key-value database and the target key-value database also requires a separate configuration script. At the same time, since the data that has changed between the source database and the target database is unknown, for the data security of the target database, it will be necessary to manually compare the differences item by item in each data table before performing the migration. Therefore, it is difficult to apply it to the data warehouse scenario of enterprises and institutions. Second, to implement services with business functions in the production environment of enterprises and institutions, in addition to the need to migrate the database, a large amount of data such as various programs and files with complex sources also needs to be migrated. The database migration script does not have the ability to migrate this data and still requires manual participation in comparison and item-by-item migration. Therefore, it is difficult to apply it to the data warehouse scenarios of enterprises and institutions.

[0039] In view of this, this specification provides a data migration solution for data warehouse scenarios applied to enterprises and institutions, which can realize a unified data migration method for multiple businesses, improve data migration efficiency, security and implementation cost, and do not require manual item-by-item comparison of differences and migration. In the solution disclosed in this specification, the data to be migrated in the source task environment can be divided according to task classification to obtain the source data of multiple task entities. The source data of the task entity corresponding to each task classification can cover all the data involved in a business function. These source data can be stored in the source data warehouse. There is no need for a third-party data warehouse or storage system (relative to the source task environment and the target task environment). The source task data can be obtained from the source data warehouse to provide the source task data corresponding to the task entity to be migrated. The source task data can then be compiled into a mirror file stored by the source server, and a mirror compilation event of the corresponding task entity to be migrated is generated. The mirror file can provide a relatively fixed migration transition environment. The corresponding source task data between different task entities to be migrated will no longer change after compilation, avoiding the occurrence of data intersection of different business functions. The mirror compilation event can provide the location information of the task entity and the migration stage and the instruction information for continuing the operation. The source task data corresponding to different task entities within various task classifications is migrated to the target server in the target task environment using a unified image file, achieving unified data migration between different environments. Data related to different business functions is not migrated separately, eliminating the need for manual data migration. This achieves high migration efficiency and low cost, and can indicate the restoration of business functions equivalent to those in the source task environment through image compilation events. It should be understood that the methods provided herein can be executed by devices with computing, instruction processing, and communication capabilities, such as servers or electronic devices.

[0040] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0041] One embodiment of the present specification provides a data migration method that can be applied to a migration server. The migration server can be deployed with a migration service. The migration service can be implemented through an operation command set script and / or application program. When the migration service is executed by the migration server, data migration between different environments can be implemented without manually comparing and migrating each data row / table or each file. The migration server can be one or more servers, which can be physical servers or server instances or container instances with communication capabilities. The hardware of the instance can be a resource instance in a server cluster composed of processor resources and memory resources and having computing and instruction processing capabilities. The resource instance can be, for example, a cloud server or a flexible application instance that uses physical server resources.

[0042] In the embodiment of this specification, when data migration is required for the target task environment, the data to be migrated can be prepared in the source task environment. Figure 1 , the data migration method may include:

[0043] S1. Acquire source task data from a source data warehouse. The task data stored in the source data warehouse includes source data of multiple task entities. The source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment.

[0044] In some possible implementations, the various businesses of enterprises and institutions can be regarded as different data migration tasks, and various businesses can correspond one-to-one with tasks, and each business classification also corresponds one-to-one with a task classification. The task environment can be a software environment that performs tasks corresponding to the various businesses of enterprises and institutions. The software environment can include one or more computer systems and computer programs that support communication between programs or between systems. In some possible application scenarios, the task environment can also include a hardware environment that provides support for the aforementioned software environment. For example, the hardware environment can include equipment that supports communication networks and equipment that provides computing and instruction processing capabilities. The source task environment and the target task environment can be two task environments logically distinguished by data migration. The source task environment can be a task environment that provides data to be migrated, and the target task environment can be a task environment that receives migrated data. For example, the source task environment can be a test environment, the target task environment can be a production environment, the target task environment and the source task environment can be the same enterprise or two different enterprises (the client and the deliverer); for another example, the production environment that serves as the target task environment can also serve as the source task environment. In this case, the source task environment can be the production environment, and the target task environment can be a backup production environment. The target task environment can also be a test environment that is a mirror image of the same tasks as the production environment, that is, the source task environment can be a mirror environment with the same tasks as the target task environment, so as to provide a basis for application innovation and update iteration of enterprises and institutions.

[0045] In some possible examples, for software development enterprises and institutions, multiple services may include data quality auditing, script development, data access, workflow, etc. Each service may be used to implement a corresponding business function.

[0046] In the first business example, data quality audit can realize the data quality audit function. This business function can be implemented using data collection tools, database comparison scripts (SQL scripts), data analysis tools, notification sending and receiving programs, etc., and will involve the application of the tool and the dependency / mapping relationship library, the database that stores structured business data, the data to be processed in different tools (in folders or caches) with different data formats, script files containing unstructured data, notification programs, etc. The data quality audit function can be used to check the standardization, completeness and consistency of business data and notify designated personnel of changes in data quality levels, etc., and can evaluate the software code standardization and functional effectiveness of enterprises and institutions.

[0047] In the second business example, script development can realize the script development function. This business function can be implemented using development tools (such as IDE tools), applications to be tested, testing tools, etc., and will involve the development of semi-finished scripts / scripts (SQL, C#, Shell, Python, Go and other computer languages), test data, message queue components, data tables specified in the scripts, as well as data mapping rule files within the project, specified encoding files, data conversion files, etc. The script development business function can be used to deliver new functional applications and maintain updated iterative applications, and can be the source of the main products provided by enterprises and institutions.

[0048] In the third business example, workflow can realize workflow functions. This business function can be implemented using a workflow system, which will involve code management warehouses, process management data (process label resources, process node resources, etc.), dialogue interaction programs, code development version data and modification records of each project, etc. The workflow business function can be used for information exchange between people involved in development, transfer code files, documents, etc., and can provide enterprises and institutions with visual, pipelined product development process data and product release data.

[0049] In the three business examples above, we can observe that the software data within the task environment can include a wide variety of data formats, multiple script file types, unstructured and structured business data, multiple interface data, multiple databases / tables, multiple business applications, and tools. This data migration is massive, numerous, and requires complex data mapping. Manual data migration is extremely difficult. It should be noted that the aforementioned enterprises and institutions can encompass a wide range of business scopes and are not a single implementation method. For example, they can also include advertising, media, social media, and other enterprises and institutions.

[0050] In the embodiment of the present specification, the data to be migrated may be divided before the aforementioned step S1. The aforementioned data migration method may further include:

[0051] C1: Determine the task categories of the source task environment and the data to be migrated for each task category.

[0052] In some possible implementations, the corresponding task classification can be determined according to the aforementioned business classification, and the unique identifier of the task classification can be used to mark the data to be migrated in the source task environment, thereby determining the data to be migrated under each task classification. In some possible examples, if the identifier of the data quality audit classification is 001 and the identifier of the script development classification is 002, then the application program and dependency / mapping relationship library involved in the tool, the database storing structured business data, the data to be processed in different data formats in different tools (in folders or caches), the script file containing unstructured data, the notification program, etc. can be attached with the identifier 001, and the data to be migrated involved in the aforementioned script development can be attached with the identifier 002, for example, the identifier can be attached to the database export file name, the file name of various non-database (programs, scripts, mapping / conversion rule files, etc.) data to be migrated, and optionally, the non-database exported data to be migrated after the identifier is attached can be written to a specified database (or file storage system), and then the identifier of the non-database exported data to be migrated is attached to the data exported from the specified database, which is conducive to further improving efficiency. In other possible examples, the data involved can be determined based on the unique identifier of the personnel under the business classification, and the same identifier (such as the script development department identifier) ​​can be uniformly added to the data under the same business classification, while the identifiers of the data under different business classifications can be different. This can form structured data, and the aforementioned division method can also include:

[0053] C2, based on the task entity structure and the data to be migrated of each task category, forms the task entity and source data corresponding to each task category.

[0054] In some possible implementations, an entity can be structured data that carries a unique identifier and an encapsulation format. A task entity can be structured data that encapsulates source data under a task classification. The source data can include data to be migrated and / or a data description. The source data of a task entity can include the task entity's unique identifier and corresponding value, and a data description and corresponding value. The data description can be used to query the task classification or the data to be migrated. The data description can include any combination of at least three of the task entity's name, type (based on the task classification), creation and update time, version number, the name of the database involved, the name of the data table, and the content of the task entity. The choice of data description can vary between different task entities. The content of the task entity can include the data to be migrated or the storage address of the data to be migrated, or the script content for pulling the data to be migrated. For example, the data to be migrated can include a project number, project name, data source identifier, keywords, and data pulled through the address and script content. The storage address of the data to be migrated can include the address of a database export file or the file address of a non-database export file. The script content for pulling the data to be migrated can include a command and command parameters for exporting the database to a specified file or a command and command parameters for transferring a file of a non-database export file. The corresponding value type can include string and numeric type, and the corresponding value type can also include date; the unique identifier of the task entity can be selected and combined based on the entity feature identifier, for example, the entity feature identifier can include personnel identifier, department identifier, and project identifier. In some possible examples, a task entity structure corresponding to a divided task classification can be written as:

[0055] {

[0056] Unique ID: The unique ID of the task entity,

[0057] Name: the name of the task entity,

[0058] Type: Task Classification,

[0059] Creation time: the creation time of the task entity,

[0060] Update time: the update time of the task entity,

[0061] Content: the content of the task entity,

[0062] Version number: the version number of the task entity,

[0063] Database name: the name of the database involved in the task entity.

[0064] Data table name: the name of the data table involved in the task entity

[0065] }

[0066] For example, Figure 2Under the data quality audit category, a task entity may include the unique identifier of the task entity and the corresponding string value, name and the corresponding string value, type and the corresponding string value (data quality audit category), data source identifier (such as the number identifier of the database that can be identified by the database driver file) and the corresponding numeric value, database name and the corresponding string value, data table name and the corresponding string value. Under the script development category, a task entity may include the unique identifier of the script development category and the corresponding string value, name and the corresponding string value, type and the corresponding string value (script development category), creation time and the corresponding date, script content and the corresponding string value. Under the data access category, a task entity may include the unique identifier of the data access category and the corresponding string value, name and the corresponding string value, type and the corresponding numeric value (data access category), creation time and the corresponding date. Under the workflow category, a task entity may include the unique identifier of the task entity and the corresponding string value, name and the corresponding string value, type and the corresponding string value (workflow category), and keyword (key, which can be used to locate information of the task entity between different environments). The aforementioned data migration method may also include:

[0067] C3. Based on the entity structure of the source data warehouse and each task entity, the source data of each task entity is stored as task data in the source data warehouse.

[0068] In some possible implementations, the data warehouse can be a data management system, and the task data stored in the source data warehouse can be derived from the source data of the task entity pulled from the source data warehouse, or the source data of the task entity can be pulled into the source data warehouse via the aforementioned migration server. The source data of the task entity can be stored as task data in the format of the entity structure of the source data warehouse. In some possible examples, the entity structure of the source data warehouse corresponding to a task entity can be written as:

[0069] {

[0070] Unique ID: The unique ID of the task entity,

[0071] Name: the name of the task entity,

[0072] Type: Task Classification,

[0073] Level: Level name,

[0074] Content: The content of the task entity

[0075] }

[0076] In the entity structure of the source data warehouse, the unique identifier of the task entity can serve as the unique identifier of the task data, or a separate unique identifier of the task data can be generated that corresponds one-to-one with the unique identifier of the task entity. The hierarchy represents the level at which the source data is processed in the source task environment, such as the preprocessing level, data model processing level, and completion processing level. This hierarchy is used to record processing hierarchy information to ensure consistency of the processing hierarchy information of the source data after recovery. The task data in the source data warehouse can include the content of the task entity to pull the corresponding data to be migrated for the task entity under each task category.

[0077] In the embodiment of the present specification, in the aforementioned step S1, when data migration is required, the task entities under the task categories configured by default can be automatically migrated according to the task categories configured by default. In some possible implementations, all or part of the aforementioned task categories (script development categories and data access categories) can be used as the task categories configured by default, and when migrating data, task entities of the script development categories and data access categories are automatically obtained from the source data warehouse. In other possible implementations, in order to achieve the migration of data corresponding to the specified business functions and increase scalability, the task categories can be manually specified and selected, and the aforementioned step S1 may include:

[0078] S101: Determine a unique identifier of a task entity to be migrated based on a selected task classification.

[0079] In some possible examples, the task entities under each task category may have been classified and grouped. For example, task entities under the same task category may be placed in the same list. After selecting a task category, the task entities to be migrated under the task category may be determined to achieve unified migration of data corresponding to business functions without having to compare differences item by item for task entities under the same task category. The aforementioned step S1 may also include:

[0080] S102: Query the unique identifier in the source data warehouse, and obtain source task data through identifier matching.

[0081] In some possible examples, the source task data corresponding to the source data of the task entity can be obtained by matching the unique identifier of the task entity in the task data stored in the source data warehouse. The source task data can be the data entity to be migrated.

[0082] In the embodiment of this specification, the aforementioned migration server may have the same data processing method for task entities under the same task category. Before compiling the source task data, the aforementioned data migration method may further include:

[0083] P0: Determine a data migration rule for migrating the task entity to be migrated from the source task environment to the target task environment based on the task classification and data migration rule structure corresponding to the task entity to be migrated.

[0084] The data migration rules are used to indicate the rules and parameter information of data processing operations; the data processing operations include image compilation operations, data migration operations, and data recovery operations performed on the source task data.

[0085] In some possible implementations, the data migration rule structure may include a rule module, a rule type, parameter information, a rule parameter type, and the like. The rule module may be used to indicate the number of the task classification corresponding to the current data migration rule. The rule type may include a simple type and a complex type, which are used to indicate the classification information of the current data migration rule respectively. The classification information may query the command (simple type, which can be queried from the preset command table) or the script file (complex type, which can obtain the corresponding script file from the preset script library) corresponding to the task entity to be migrated. The parameter information may be used to indicate the parameter information that needs to be loaded for the command or script file. The rule parameter type may be used to indicate the data type of the loaded parameter information. In some possible examples, the task classification may include workflow classification (1), data access classification (2), data export classification (3), script development classification (4), semi-structured storage classification (5), database synchronization (e.g., data change capture, Change Data Capture, CDC) classification (6), and data quality audit classification (7). The data migration rule structure corresponding to the task classification at this time may be written as:

[0086] {

[0087] Rule module: {

[0088] 1: Workflow scheduling data,

[0089] 2: Data access,

[0090] 3: Data export,

[0091] 4: Script development,

[0092] 5: Semi-structured data,

[0093] 6: CDC data,

[0094] 7: Data quality audit data

[0095] },

[0096] Rule Type: {

[0097] Simple type: [project code, project name, data source, database name, message queue], complex type: [audit task data, export task data, import task data]

[0098] },

[0099] Parameter information: {

[0100] IDE script: {project coding},

[0101] Audit: {project code, database, resource code, unique identifier of task entity},

[0102] SQL script: {data source identifier},

[0103] Sub-process: {project code, project name},

[0104] Data import: {project code, project name, unique identifier of task entity},

[0105] Data export: {project code, project name, unique identifier of task entity},

[0106] Database: {namespace, database name},

[0107] queue: {queue priority},

[0108] Project: {project code, project name}

[0109] },

[0110] Rule parameter type: {

[0111] Data type: string / numeric / function call

[0112] }

[0113] }

[0114] In the data migration rule structure, parameter information can be selected from one or a combination of parameters and applied to the processing method (command or script file) corresponding to the rule type. The data migration rule structure can be configured based on the task environment. Based on the data in the source task environment and the data migration rule structure, multiple data migration rule files corresponding to the rule module numbers can be generated. The data migration rule files can be stored on the migration server or in a designated storage location in the source task environment, which can be accessed by the migration server.

[0115] In some possible examples, the aforementioned migration server may be deployed with a migration service, which may have the function of data processing operations, that is, it may have the functions of image compilation operations, data migration operations, and data recovery operations. In the task classification example of the default migration configuration, the migration server may not need to determine the data migration rules, that is, the rules and parameter information may have been configured on the migration server. At this time, the migration server can compile the source task data and migrate the compiled image file to the target server. In the example that is conducive to the flexible migration of business functions, the migration server can also compile the source task data corresponding to the task entity to be migrated based on the determined data migration rules. In the embodiment of this specification, the aforementioned data migration method may also include:

[0116] S2: Compile the source task data into an image file stored by the source server, and generate an image compilation event of the task entity to be migrated corresponding to the source task data.

[0117] In some possible implementations, the aforementioned migration server may perform compilation through an image compilation operation, and the image compilation operation may include a file writing operation and a compression operation. The aforementioned step S2 may include:

[0118] S201 , through the image compilation operation, writing the source task data and the data migration rules into files corresponding to the task classifications respectively.

[0119] The task classification may be a selected task classification or a default configured task classification, and the data migration rule may be a file configured in the migration server, or a file determined by the aforementioned data migration rule. The data migration rule may provide instruction information for the image compilation operation, so as to write the current source task data to a specified file by command or script file and by task classification. The writing method may be copying, and the specified file may include, for example, a folder, a text file, a binary file, a database export file, etc. The aforementioned step S2 may also include:

[0120] S202: Compress the written file into a mirror file through the mirror compilation operation, and store the mirror file in the source server.

[0121] In some possible examples, the image file can be a compressed data package. Before migration, the source data corresponding to different business functions is in a fixed transition environment. Therefore, the source data corresponding to different business functions can be uniformly migrated to the target task environment. The source server and target server can be data objects used to store files. The entity structure of the source server and target server can be written as:

[0122] {

[0123] Unique ID: The unique ID of the image file.

[0124] Name: the name of the stored image file

[0125] Path: the path of the stored image file,

[0126] Storage time: Date

[0127] }

[0128] In some possible examples, the source server and the target server may also be servers of the same type as the migration server. In this case, the image file may be a compressed data package of the written file; the image file may also be an image backup file of the entire source server, which may include the aforementioned compressed data package or the written file. The compiled mirror file carries the mapping information between the source data of the task entity to be migrated in the source task environment and the source task data in the source data warehouse, as well as the mapping information between the task data in the file to be compiled by the mirror file in the source task environment (i.e., the aforementioned file after writing) and the source task data in the source data warehouse. At the same time, the mirror file also carries the mapping information between the task data in the file decompiled by the mirror file in the target task environment and the target task data in the target data warehouse, as well as the mapping information between the target task data in the target data warehouse and the target data of the task entity in the target task environment, so that the mapping information is provided in the mirror environment, wherein the mapping information may include the correspondence information between the data to be migrated in the source data warehouse and the task entity and the correspondence information between the data in the file after writing or decompiled and the task entity, such as the correspondence information of the identifier (including keywords), the correspondence information of the network address, the correspondence information of the file path, etc. The decompiled file can be consistent with the aforementioned file after writing. The operations of compilation / compression and decompilation / decompression can keep the relationship between files unchanged, such as the folder hierarchy, the link relationship between files, etc. After obtaining the image file, an event record may be generated. The aforementioned step S2 may further include:

[0129] S203 : Based on the image compilation event structure, image compilation events of the task entities to be migrated corresponding to the source task data are generated respectively through the image compilation operation.

[0130] For some possible examples, please combine Figure 3 In order to locate the task entity that has errors during data migration and observe the migration progress of the task entity, the event structure generated by each data processing operation can be written as:

[0131] {

[0132] Unique ID: The unique ID of the task entity,

[0133] Name: the name of the data processing operation,

[0134] Type: the type of data processing operation,

[0135] Data source: the identifier of the source data warehouse,

[0136] Data destination: ID of the target data warehouse,

[0137] Start time: the start time of the event,

[0138] End time: the end time of the event,

[0139] Status: The execution status of the task data,

[0140] Log information: Migration log of task entity

[0141] }

[0142] In this event structure, the name of the data processing operation can include the name of the image compilation operation, data migration operation and data recovery operation, the type of the data processing operation can include the number of the image compilation operation, data migration operation and data recovery operation (such as operation 1, operation 2, operation 3), and the execution status of the task data can include a success status (such as a value of 1) and a failure status (such as a value of 0).

[0143] It should be noted that the migration server can have data read and write permissions for the target task environment and the source task environment. The target task environment can include a target server, and the source task environment can include a source server. The source server and the target server can be servers with the same entity structure, or the target server can be a server that is a mirror backup of the source server. The target task environment and the source task environment can both include a data warehouse, and the target task environment can include a target data warehouse, and the entity structure of the data warehouse can be the same. The target task environment and the source task environment can also be mirror environments with the same task entity structure, for example, the task entity structures of the task entities between the two environments have the same type and the same number of task entities of any type. If the task environment that is the target task environment needs to be changed to the source task environment, the aforementioned target task environment can also include the same data migration rule structure as the aforementioned source task environment. The specific implementation methods of the above-mentioned data migration rule structure, arbitrary entity structure, and arbitrary event structure are all examples, and are not limited implementation methods. They can be selected and designed based on different environments and businesses in the application scenario, as well as testing and usage effects.

[0144] In the embodiment of the present specification, after the aforementioned image compilation event is generated, the image file can be migrated based on the instruction of the image compilation event. The aforementioned data migration method can also include:

[0145] S3: Based on the image compilation event, migrate the image file stored in the source server to the target server, and generate an image migration event for the task entity to be migrated. The image migration event is used to indicate that the image file stored in the target server is restored to the target task environment.

[0146] In some possible implementations, the image compilation event can provide the migration server with information on the execution status of the task entity. If the execution status in the image compilation event is a failed state, the migration server can temporarily ignore the task entity and return the unique identifier of the task entity that reported the failed state after the migration is completed, so as to facilitate locating the task entity and the corresponding source data from a variety of large-scale data. The migration server can migrate the compressed image file. The aforementioned step S3 may include:

[0147] S301 : When the execution status in the image compilation event is a success status, the image file stored in the source server is transferred to the target server through the data migration operation.

[0148] In some possible examples, the data migration operation can complete the data migration between different environments by transmitting a compressed data package between the source server and the target server. There is no need to transmit each different data item separately. Instead, the migration transition environment of the task entity corresponding to the business function is transmitted to the target server in a unified file format to maintain data integrity and consistency. After the image file migration is completed, an image migration event can be generated. The aforementioned step S3 can also include:

[0149] S302 : Based on the mirror migration event structure, generate mirror migration events of the task entities to be migrated respectively through the data migration operation.

[0150] In some possible examples, the image migration event structure may be the aforementioned event structure. If the execution status in the image migration event is a failure, the migration server may retry step S302. If the number of retries reaches a threshold, the migration server may return a data migration operation failure message. The failure message may include, for example, network failure, target server non-response, or incomplete data packet transmission.

[0151] In the embodiments of this specification, the aforementioned mirror file can be used as a backup file of the task entity and the corresponding source data in the target task environment in the source task environment, and can also be used for data recovery in the target task environment. The aforementioned data migration method may also include:

[0152] S4. Obtaining a selected or default mapping relationship. The mapping relationship includes a data correspondence relationship and a correspondence relationship between the source data warehouse and the target data warehouse in the target task environment. The data correspondence relationship includes a correspondence relationship between the source task data and the target task data. The source data warehouse and the target data warehouse have the same physical structure. The network address of the target data warehouse, such as a domain name address or an IP address, can be selected.

[0153] In some possible implementations, the default mapping relationship can be provided by the aforementioned image file, and the decompiled file can carry the aforementioned mapping information and serve as the default mapping relationship. In order to facilitate the flexibility and scalability of the data recovery function and avoid the unavailability of the default designated data warehouse, you can choose to manually specify the target data warehouse and target task data in the target task environment, and establish corresponding relationships with the source data warehouse and source task data respectively, which can include hierarchical relationships between databases, link relationships between non-database files, etc. In some possible examples, the data recovery operation may include decompression operations or decompilation operations. Please combine Figure 4 The aforementioned data migration method may further include:

[0154] S5. When the execution status in the image migration event is a success status, the image file stored on the target server is decompressed or the server image backup file is decompiled through the data recovery operation to obtain the data migration rules and the task data in the decompressed file or the decompiled file as the target task data of this migration.

[0155] The task data may be consistent with the source task data in the aforementioned written file. In some possible examples, the data recovery operation may further include a parsing rule operation and a data replacement (copy) operation, and the aforementioned data migration method may further include:

[0156] S6. Based on the data correspondence and the data migration rule, the original task data in the target data warehouse is replaced with the target task data of this migration through the data recovery operation.

[0157] The data migration rules may provide instruction information on the data processing method that matches the task data in the decompressed file or decompiled file, load the instruction information through the data recovery operation, and import the data correspondence relationship for the data recovery operation, and then complete the data replacement between the decompressed file or decompiled file and the target data warehouse. The aforementioned data migration method may also include:

[0158] S7, based on the task data of this migration in the target data warehouse and the task entity structure in the target task environment, through the data recovery operation, the task data in the target data warehouse are written into the task entities corresponding to the selected or default configured task classification.

[0159] The data recovery operation may have loaded the instruction information and imported the data correspondence. The data recovery operation may be performed between the target data warehouse and each task entity, and according to the task entity structure of each task entity, the task data in the target data warehouse may be written to the task entity corresponding to the task classification. In order to facilitate the efficiency of the aforementioned copying and replacing of data, a unique keyword that can be queried by the migration server may be configured in each task entity. In the source task environment and the target task environment that are mirror images of each other, the data copying and replacement may be completed by querying the keyword. After the data is replaced, the data recovery operation may generate a mirror recovery event. The aforementioned data migration method may also include:

[0160] S8. Based on the mirror recovery event structure, generate a mirror recovery event for the task entity of this migration through the data recovery operation.

[0161] The image recovery event structure may be the aforementioned event structure. In some possible examples, if the execution status recorded in the image recovery event is all successful and each task classification is migrated by default, then the task entity under the aforementioned data access classification, the task entity under the script development classification, the task entity under the data quality audit classification, and the task entity under the workflow classification can be obtained in the target task environment. After the aforementioned data migration is completed, if the execution status of any event is a failed state, the unique identifier and / or task classification of the task entity in any of the events can be extracted and recorded. Any of the events may include an image compilation event, an image migration event, or an image recovery event, thereby providing data migration with location information of the task entity and the task classification, as well as location information of the failed stage of the data processing operation, so as to facilitate the data migration of the task classification or task entity to be attempted separately, without the need for full migration again, thereby further improving the efficiency of data migration, such as Figure 5 In the task list, select each task entity under different task categories through the selection window.

[0162] In addition, you can also Figure 5In the selection window in step S1, before step S1, select the task categories and task entities for this migration. For example, the selection window may include a "Select All" button for migrating all task categories. The current selection window presents a list of candidate forms, and the selected form can record the task categories and task entities that have been selected for migration. In the operation area, there may be an operation button "Task Selection" for the list of candidate forms. The data corresponding to the name of the task entity that has been selected with the √ mark will be selected for this migration. The operation area may also include an input box for filtering date and name queries, as well as a "Query" button. In the list of candidate forms, the name of each task entity can be combined with the data source identification field (identifiers 1 to 7), the synchronization type field (which can include database and non-database types and can be used to provide the classification basis for the aforementioned rule types), the time field, the associated personnel field (personnel identification 1 to 3), and the field of the "Select" operation button. The name of the task entity can include the name of any data that needs to be migrated in any application scenario, such as parameter tables, access tables 1 to 2, test scripts, and unstructured data. The selection window can be presented on a device terminal that communicates with the migration server. The device terminal can include a computer, mobile electronic device, etc. The device terminal can send the task category and task entity name selected by the technician to the migration server. The migration server can then query the source task data corresponding to the task entity in the source data warehouse and start the data migration. In this example, the technician can automatically migrate data by simply selecting the task category and task entity, without having to compare data differences and migrate each item one by one.

[0163] It should be added that the aforementioned migration server can generate a graph structure (such as Figure 4 , Figures 2 to 4 The ellipsis in the middle indicates that more customized data can be included). This allows technicians to locate the stage of the data in the migration process and the execution status of the data in each stage.

[0164] In the embodiment of this specification, a unified migration method between multiple businesses is implemented between the source task environment and the target task environment, which has the characteristics of high migration efficiency and reduces the data migration implementation cost for users, while providing long-term evolution support for the application innovation and update iteration of enterprises and institutions. The source task environment and the target task environment, which are mirror images of each other, can provide a basis for further improving migration efficiency. The mirror file can provide a data transition environment before and after migration and carry mapping information. It can automatically complete unified data migration without manual intervention, and restore multiple business functions in the source task environment in the target task environment. For example, the business functions corresponding to the task entities built in the test environment can be migrated to the production environment for use. The source data of the task entity can be quickly traced back to the stage in the entire data migration process through the graph structure, and the task data that needs to be migrated can be manually selected for migration, which has the flexibility of data migration. After the migration is completed, the execution status of the data migration can be observed, whether it is a successful state or a failed state, and a retry can be performed for the data in the failed state. It can be applied to task environments of multiple data source types and multiple task types, and can automatically perform data processing operations such as backup, migration and recovery.

[0165] The embodiment of this specification also provides a data migration system based on the same inventive concept as the above embodiment, please refer to Figure 6 The data migration system 600 may include:

[0166] An acquisition module 601 is configured to acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, wherein the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment;

[0167] An image compilation module 602 is configured to compile the source task data into an image file stored by a source server, and generate an image compilation event for a task entity to be migrated corresponding to the source task data;

[0168] The image migration module 603 is used to migrate the image file stored in the source server to the target server based on the image compilation event, and generate an image migration event for the task entity to be migrated. The image migration event is used to restore the image file stored in the target server to the target task environment.

[0169] Optionally, combined Figure 7 The data migration system 600 may further include a partitioning module 604, which may be configured to:

[0170] Determine the task categories of the source task environment and the data to be migrated for each task category;

[0171] Based on the task entity structure and the data to be migrated of each task category, a task entity and source data corresponding to each task category are formed;

[0172] Based on the entity structure of the source data warehouse and each task entity, the source data of each task entity is stored as task data in the source data warehouse.

[0173] Optionally, obtaining source task data from a source data warehouse includes:

[0174] Based on the selected task classification, determine the unique identifier of the task entity to be migrated;

[0175] The unique identifier is searched in the source data warehouse, and the source task data is obtained through identifier matching.

[0176] Alternatively, refer to Figure 8 The data migration system 600 may further include a data operation module 605, which may be used to:

[0177] Based on the task classification and data migration rule structure corresponding to the task entity to be migrated, a data migration rule for migrating the task entity to be migrated from the source task environment to the target task environment is determined; wherein,

[0178] The data migration rules are used to indicate the rules and parameter information of the data processing operation;

[0179] The data processing operation includes an image compilation operation, a data migration operation, and a data recovery operation performed on the source task data.

[0180] Optionally, compiling the source task data into an image file stored by the source server and generating an image compilation event for the task entity to be migrated corresponding to the source task data includes:

[0181] By means of the image compilation operation, the source task data and the data migration rules are respectively written into files corresponding to the selected task classification;

[0182] Compressing the written file into a mirror file through the mirror compilation operation, and storing the mirror file in the source server;

[0183] Based on the image compilation event structure, image compilation events of the task entities to be migrated corresponding to the source task data are generated respectively through the image compilation operation.

[0184] Optionally, based on the image compilation event, migrating the image file stored in the source server to the target server and generating an image migration event for the task entity to be migrated include:

[0185] When the execution status in the image compilation event is a success status, the image file stored in the source server is transferred to the target server through the data migration operation;

[0186] Based on the mirror migration event structure, the mirror migration events of the task entities to be migrated are generated respectively through the data migration operation.

[0187] Alternatively, as Figure 9 The data migration system 600 may further include: an image recovery module 606, wherein the image recovery module 606 may be used to:

[0188] Acquire a selected or default mapping relationship, wherein the mapping relationship includes a data correspondence relationship and a correspondence relationship between the source data warehouse and the target data warehouse in the target task environment, wherein the data correspondence relationship includes a correspondence relationship between the source task data and the target task data;

[0189] The source data warehouse and the target data warehouse have the same entity structure.

[0190] Optionally, the image recovery module 606 may also be used to:

[0191] When the execution status of the image migration event is a success state, decompressing the image file stored on the target server through the data recovery operation to obtain the data migration rule and the task data in the decompressed file as the target task data of this migration;

[0192] Based on the data correspondence and the data migration rules, the original task data in the target data warehouse is replaced with the target task data of this migration through the data recovery operation;

[0193] Based on the task data of this migration in the target data warehouse and the task entity structure in the target task environment, the task data in the target data warehouse are written into the task entities corresponding to the selected task classification through the data recovery operation;

[0194] Based on the mirror recovery event structure, a mirror recovery event of the task entity of this migration is generated through the data recovery operation.

[0195] Alternatively, as Figure 10 As shown, the data migration system 600 may further include a recording module 607, which may be used to:

[0196] If the execution status of any event is a failure status, the unique identifier and / or task classification of the task entity in the any event is extracted and recorded.

[0197] Optionally, wherein

[0198] The target task environment includes a target data warehouse;

[0199] The target task environment and the source task environment are mirror images of the same task entity structure;

[0200] The mirror file carries mapping information between source data of the task entity to be migrated in the source task environment and source task data in the source data warehouse, and mapping information between task data in the file to be compiled by the mirror file in the source task environment and source task data in the source data warehouse;

[0201] The image file also carries mapping information between the task data in the file decompiled by the image file in the target task environment and the target task data in the target data warehouse, as well as mapping information between the target task data in the target data warehouse and the target data of the task entity in the target task environment.

[0202] The embodiments of this specification also provide an electronic device under the same inventive concept as the above embodiments, which may include: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method in the above embodiments by executing the instructions stored in the memory. Please refer to Figure 11 , provides an exemplary electronic device, the internal structure of which can be as shown in FIG. Figure 11 As shown, it can be a server, an industrial computer, a user device, a terminal device, a microcontroller, etc. The electronic device includes a processor A01, a network interface A02, and a memory connected via a bus. The processor A01 of the electronic device is used to provide computing, instruction processing, and control capabilities. The memory of the electronic device includes a memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01 and a computer program B02. The memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the electronic device is used to communicate with the network. When the computer program B02 is executed by the processor A01, the method in the aforementioned embodiment is implemented.

[0203] In one embodiment of the present specification, when the computer program B02 is executed by the processor A01, the implemented method may include:

[0204] Acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, and the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment;

[0205] Compiling the source task data into an image file stored by a source server, and generating an image compilation event for a task entity to be migrated corresponding to the source task data;

[0206] Based on the image compilation event, the image file stored in the source server is migrated to the target server, and an image migration event of the task entity to be migrated is generated. The image migration event is used to instruct to restore the image file stored in the target server to the target task environment.

[0207] An embodiment of this specification also provides a machine-readable storage medium under the same inventive concept as the aforementioned embodiment, which stores machine instructions. When the machine instructions are run on a machine, the machine executes the method in the aforementioned embodiment. The machine may include a computer and a device with communication, computing and instruction processing capabilities.

[0208] In one embodiment of this specification, when the machine instructions are executed on a machine, the method executed by the machine may include:

[0209] Acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, and the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment;

[0210] Compiling the source task data into an image file stored by a source server, and generating an image compilation event for a task entity to be migrated corresponding to the source task data;

[0211] Based on the image compilation event, the image file stored in the source server is migrated to the target server, and an image migration event of the task entity to be migrated is generated. The image migration event is used to instruct to restore the image file stored in the target server to the target task environment.

[0212] The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this specification can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this specification can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0213] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0214] In short, the above description is only a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in the scope of protection of this specification.

[0215] The systems or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer.

[0216] Machine-readable storage media can be computer storage media, which can include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0217] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0218] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A data migration method, characterized in that: This data migration method includes: Acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, and the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on task classification of the source task environment; Compiling the source task data into an image file stored by a source server, and generating an image compilation event for a task entity to be migrated corresponding to the source task data; Based on the image compilation event, the image file stored on the source server is migrated to the target server, and an image migration event is generated for the task entity to be migrated. The image migration event is used to indicate that the image file stored on the target server is restored to the target task environment, wherein the source task data corresponding to different task entities divided into multiple task categories are all migrated to the target server as one image file.

2. The data migration method according to claim 1, characterized in that: The data migration method also includes: Determine the task categories of the source task environment and the data to be migrated for each task category; Based on the task entity structure and the data to be migrated of each task category, a task entity and source data corresponding to each task category are formed; Based on the entity structure of the source data warehouse and each task entity, the source data of each task entity is stored as task data in the source data warehouse.

3. The data migration method according to claim 1, wherein: Acquiring source task data from the source data warehouse includes: Based on the selected task classification, determine the unique identifier of the task entity to be migrated; The unique identifier is searched in the source data warehouse, and the source task data is obtained through identifier matching.

4. The data migration method according to claim 1 or 3, characterized in that: The data migration method also includes: Based on the task classification and data migration rule structure corresponding to the task entity to be migrated, a data migration rule for migrating the task entity to be migrated from the source task environment to the target task environment is determined; wherein, The data migration rules are used to indicate the rules and parameter information of the data processing operation; The data processing operation includes an image compilation operation, a data migration operation, and a data recovery operation performed on the source task data.

5. The data migration method according to claim 4, characterized in that: The step of compiling the source task data into an image file stored by a source server and generating an image compilation event of a task entity to be migrated corresponding to the source task data includes: By means of the image compilation operation, the source task data and the data migration rules are respectively written into files corresponding to the selected task classification; Compressing the written file into a mirror file through the mirror compilation operation, and storing the mirror file in the source server; Based on the image compilation event structure, image compilation events of the task entities to be migrated corresponding to the source task data are generated respectively through the image compilation operation.

6. The data migration method according to claim 5, characterized in that: The step of migrating the image file stored in the source server to the target server based on the image compilation event and generating an image migration event for the task entity to be migrated includes: When the execution status in the image compilation event is a success status, the image file stored in the source server is transferred to the target server through the data migration operation; Based on the mirror migration event structure, the mirror migration events of the task entities to be migrated are generated respectively through the data migration operation.

7. The data migration method according to claim 6, characterized in that: The data migration method also includes: Acquire a selected or default mapping relationship, wherein the mapping relationship includes a data correspondence relationship and a correspondence relationship between the source data warehouse and the target data warehouse in the target task environment, wherein the data correspondence relationship includes a correspondence relationship between the source task data and the target task data; The source data warehouse and the target data warehouse have the same entity structure.

8. The data migration method according to claim 7, characterized in that: The data migration method also includes: When the execution status of the image migration event is a success state, decompressing the image file stored on the target server through the data recovery operation to obtain the data migration rule and the task data in the decompressed file as the target task data of this migration; Based on the data correspondence and the data migration rules, the original task data in the target data warehouse is replaced with the target task data of this migration through the data recovery operation; Based on the task data of this migration in the target data warehouse and the task entity structure in the target task environment, the task data in the target data warehouse are written into the task entities corresponding to the selected task classification through the data recovery operation; Based on the mirror recovery event structure, a mirror recovery event of the task entity of this migration is generated through the data recovery operation.

9. The data migration method according to claim 8, characterized in that: The data migration method also includes: If the execution status of any event is a failure status, the unique identifier and / or task classification of the task entity in the any event is extracted and recorded.

10. The data migration method according to claim 1, wherein: in, The target task environment includes a target data warehouse; The target task environment and the source task environment are mirror images of the same task entity structure; The mirror file carries mapping information between source data of the task entity to be migrated in the source task environment and source task data in the source data warehouse, and mapping information between task data in the file to be compiled by the mirror file in the source task environment and source task data in the source data warehouse; The image file also carries mapping information between the task data in the file decompiled by the image file in the target task environment and the target task data in the target data warehouse, as well as mapping information between the target task data in the target data warehouse and the target data of the task entity in the target task environment.

11. A data migration system, characterized in that: The data migration system includes: An acquisition module is configured to acquire source task data from a source data warehouse, wherein the task data stored in the source data warehouse includes source data of multiple task entities, wherein the source data includes source data of task entities obtained by dividing the data to be migrated in the source task environment based on the task classification of the source task environment; An image compilation module, configured to compile the source task data into an image file stored by a source server, and generate an image compilation event for a task entity to be migrated corresponding to the source task data; An image migration module is used to migrate the image file stored on the source server to the target server based on the image compilation event, and generate an image migration event for the task entity to be migrated. The image migration event is used to restore the image file stored on the target server to the target task environment, wherein the source task data corresponding to different task entities divided into multiple task categories are all migrated to the target server as one image file.

12. An electronic device, characterized in that: The electronic device includes: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method of any one of claims 1 to 10 by executing the instructions stored in the memory.

13. A machine-readable storage medium storing machine instructions, which, when the machine instructions are executed on a machine, enable the machine to execute the method according to any one of claims 1 to 10.

Images