Data migration apparatus and method

Abstract

This invention relates to the field of data processing and discloses a data migration device and method. The device includes a backend import module and a frontend module. The backend import module acquires data source information from the current environment and the imported data source information within the imported data file package, and sends this information to the frontend module for visualization. Based on the user-selected current data source, it generates data source mapping information. According to the data source mapping information, it performs information transformation on the job objects in the imported job file data. Upon completion of the transformation, the job objects are stored in the database of the current environment. This invention only requires the user to upload the imported data file package. The backend import module parses the imported data file package to complete the import of the files and the replacement of mapping information within the imported files. This automatically completes the migration of data files between the test and production environments, offering rapid and flexible deployment, reducing manual steps, and improving the accuracy of imported files.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data migration apparatus and method. Background Technology

[0002] In the process of building data platform projects for financial companies such as securities and funds, a considerable number of data collection, data development tasks and workflows are configured according to different business needs. This also involves the backup and migration of these tasks and workflows in different environments. For example, these tasks and workflows that have been tested and verified in the development or testing environment need to be quickly and accurately migrated and deployed to the production environment.

[0003] The simplest way to migrate tasks and processes in different environments using existing technologies is to reconfigure them in the new environment. However, this method is time-consuming, labor-intensive, involves many steps, is prone to errors, and lacks flexibility.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a data migration device and method, which aims to solve the technical problems of existing data migration methods, such as numerous manual operation steps, susceptibility to errors, and poor flexibility.

[0006] To achieve the above objectives, the present invention provides a data migration device, which includes: a backend import module and a frontend module;

[0007] The backend import module is used to obtain import mapping information from the import data file package and read import data source information from the import mapping information;

[0008] The backend import module is also used to query all data sources in the current environment and send the queried data source information and the imported data source information to the frontend module for visualization.

[0009] The backend import module is also used to obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source information and the current data source;

[0010] The backend import module is also used to obtain import job file data in the import data file package and generate job objects based on the import job;

[0011] The backend import module is also used to perform information conversion on the job object according to the data source mapping information, and store the job object into the database of the current environment when the conversion is completed.

[0012] Optionally, the device further includes: a backend export module;

[0013] The front-end module is used to visualize the list of exportable jobs when exporting job data, and to send an export request to the back-end export module according to the job selected by the user.

[0014] The backend export module is used to obtain the job information of the data file to be exported according to the export request;

[0015] The backend export module is also used to generate an export mapping file based on the job information;

[0016] The backend export module is also used to convert the job to be exported and the export mapping file into preset format objects, and store the preset format objects to the corresponding storage directories respectively;

[0017] The backend export module is also used to compress the storage directory and generate an export data file package with a preset structure when all the data files to be exported have been converted and stored in the corresponding storage directory.

[0018] Optionally, the imported data file package further includes: imported resource files;

[0019] The import mapping information also includes: import metadata information;

[0020] The backend import module is further configured to determine the current metadata based on the current data source information and the import data source information, and generate metadata mapping information based on the import metadata information and the current metadata;

[0021] The backend import module is further configured to generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file;

[0022] Correspondingly, the backend import module is also used to perform information conversion operations on the job object based on the data source mapping information, the metadata mapping information, and the resource file mapping information.

[0023] Optionally, the file types of the data files to be exported include process files and task files;

[0024] The job information includes either first configuration information or second configuration information;

[0025] The backend export module is also used to obtain the first configuration information of the data file to be exported when the file type of the data file to be exported is a task file;

[0026] The backend export module is also used to obtain the process configuration information of the data file to be exported when the file type of the data file to be exported is a process file, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

[0027] Optionally, the device further includes: a log module;

[0028] The log module is used to generate runtime logs when the backend export module or the backend import module is running, and to send the runtime logs to the frontend module.

[0029] The front-end module is also used to visualize the execution progress of the back-end export module or the back-end import module based on the running log.

[0030] Furthermore, the present invention also provides a data migration method, the method comprising:

[0031] Obtain the import mapping information from the imported data file package, and read the import data source information from the import mapping information;

[0032] The system retrieves all data sources in the current environment and provides a visual display of the retrieved data source information and the imported data source information.

[0033] Obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source information and the current data source;

[0034] Obtain the import job file data from the import data file package, and generate a job object based on the import job;

[0035] The job object is transformed according to the data source mapping information, and the job object is stored in the database of the current environment when the transformation is completed.

[0036] Optionally, the method further includes:

[0037] When exporting job data, the list of exportable jobs is displayed visually, and an export request is sent to the backend export module according to the job selected by the user.

[0038] Obtain the job information of the data file to be exported based on the export request;

[0039] Generate an export mapping file based on the job information;

[0040] The data file to be exported and the export mapping file are converted into preset format objects, and the preset format objects are stored in their respective storage directories.

[0041] Once all the data files to be exported have been converted and stored in the corresponding storage directory, the storage directory is compressed to generate an export data file package with a preset structure.

[0042] Optionally, the imported data file package further includes: imported resource files;

[0043] The import mapping information also includes: import metadata information;

[0044] After the steps of obtaining the current data source selected by the user based on the visual display page, and generating data source mapping information based on the imported data source information and the current data source, the method further includes:

[0045] The current metadata is determined based on the current data source information and the imported data source information, and metadata mapping information is generated based on the imported metadata information and the current metadata;

[0046] Generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file;

[0047] Accordingly, the step of converting the information of the job object based on the data source mapping information further includes:

[0048] The job object is transformed based on the data source mapping information, the metadata mapping information, and the resource file mapping information.

[0049] Optionally, the file types of the data files to be exported include process files and task files;

[0050] The job information includes either first configuration information or second configuration information;

[0051] The step of obtaining the job information of the data file to be exported based on the user-selected data file includes:

[0052] When the file type of the data file to be exported is a task file, obtain the first configuration information of the data file to be exported;

[0053] When the file type of the data file to be exported is a process file, obtain the process configuration information of the data file to be exported, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

[0054] Optionally, the method further includes:

[0055] When importing or exporting data files, a runtime log is generated;

[0056] The operation logs are visualized.

[0057] This invention obtains import mapping information from the import data file package through a backend import module, and sends the import data source information read from the import mapping information to the frontend module for visualization. It then obtains the current data source selected by the user based on the visualization page, and generates data source mapping information based on the current data source and data source mapping information. A job object is generated based on the import job file data obtained from the import data file package. Information transformation is performed on the job object according to the data source mapping information, and upon completion of the information transformation, the job object is stored in the database of the current environment to complete the data migration. By importing data files and replacing the data mapping relationships in the imported files through the backend import module, users only need to upload the import data file package. The backend import module parses the data, sends the import data source information and all data source information in the current environment to the visualization interface of the frontend module, and then allows the user to select the data source in the current environment. This completes the import of the files and the replacement of the data mapping information in the imported files. The program automatically performs the file import and mapping information replacement, completing the migration of data files between the test and production environments. This results in rapid and flexible deployment, reduces manual operation steps, and improves the accuracy of imported files. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the structure of the first embodiment of the data migration device proposed in this invention;

[0060] Figure 2 This is a flowchart of the core logic processing of the backend import module of the data migration device of the present invention;

[0061] Figure 3 This is a flowchart of the core logic processing of the backend export module of the data migration device of the present invention;

[0062] Figure 4 This is a schematic diagram of an application scenario of the data migration device of the present invention;

[0063] Figure 5 This is a flowchart illustrating the first embodiment of the data migration method of the present invention;

[0064] Figure 6This is a flowchart illustrating a second embodiment of the data migration method of the present invention.

[0065] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0066] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0068] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, the user should consider such a combination of technical solutions to be non-existent and not within the scope of protection claimed by this invention.

[0069] This invention provides a data migration device that can be applied to data platform-related projects to perform batch and automated data migration of data configured with various data tasks and data operation processes on the platform. Data platforms typically employ big data technologies, providing functions such as data acquisition, data development, data standards, data quality, data assets, and data security. They can meet the business scenarios of financial institutions, including big data platforms, large-scale data processing, real-time / near real-time computing, and financial data warehouses.

[0070] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the first embodiment of the data migration device proposed in this invention;

[0071] based on Figure 1 The first embodiment of the data migration device of the present invention is presented.

[0072] In this embodiment, the data migration device is used to migrate data files and modify mapping information in different environments. The data migration device includes a backend import module and a frontend module.

[0073] The backend import module is used to obtain import mapping information from the import data file package and read import data source information from the import mapping information;

[0074] The backend import module is also used to query all data sources in the current environment and send the queried data source information and the imported data source information to the frontend module for visualization.

[0075] It is understandable that data migration requires exporting and importing the data to be migrated. The data to be migrated is exported from the old environment to generate an import data file package that can be imported into the new environment. It is easy to understand that the import data file package can include all or part of the information in the original environment, and the imported files can be files selected by the user in the old environment according to their needs. This embodiment does not impose any restrictions on this.

[0076] It should be noted that the aforementioned front-end module can have a visual display interface, which can include buttons for user interaction or pages for viewing, such as a selection of the import data file package, a dynamically refreshed import progress log, and an import confirmation button. Users can begin importing files by selecting the import data file package or by dragging and dropping the import data file package directly into the front-end module.

[0077] Understandably, users can upload and import data file packages through the front-end module. When the front-end module receives the imported data file package selected by the user, it can send the imported data file package to the back-end import module to start the file import.

[0078] It should be noted that the aforementioned import data file package may include import resource files and import mapping information. The import mapping information is stored in the mapping directory of the import data file package and contains mapping information of the data in the import resource files. When the backend import module obtains this mapping information, it can perform mapping replacement or generate new mapping information based on the mapping information.

[0079] It should be noted that the aforementioned imported resource files are stored in the package directory of the imported data file package, and may include dependency files, script files, etc. This embodiment does not impose any restrictions on this.

[0080] It is understood that the data source is a database or file system capable of storing data, such as the metadata database or files in various business systems, or the underlying storage (RDBMS / HDFS / MPP / OLAP / MQ, etc.) of the data platform itself. This embodiment does not impose any restrictions on this. The imported data source information may include information related to the data source, such as the data source ID and the data source name.

[0081] It should be noted that while the backend import module sends the imported data source information to the frontend module, it also returns all data source information in the current environment to the frontend module and displays it through a visual interface. Users can select the data source to use in the current environment based on the imported data source information and the data source information in the current environment.

[0082] In the specific implementation, the backend import module can obtain the import mapping information in the import data file package, and send the import data source information read from the import mapping information to the frontend module for visualization.

[0083] The backend import module is also used to obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source information and the current data source;

[0084] It is understandable that the above current data source is the data source in the current environment selected by the user based on the imported data source information.

[0085] It should be noted that the above current data source mapping information is determined based on the imported data source information and the current data source.

[0086] In the specific implementation, the backend import module obtains the current data source selected by the user based on the visualization display page, and generates data source mapping information based on the current data source and the imported data source information.

[0087] The backend import module is also used to obtain import job file data in the import data file package and generate job objects based on the import job;

[0088] It is understandable that when the imported data file package is sent to the backend import module, the backend import module can download the files in the imported data file package to the current environment and generate local files. It can also obtain the import job file data in the imported data file package and generate job objects based on the import job file data. The attributes of the job objects can include data sources, metadata, and data mapping of resource files.

[0089] It should be explained that the job types for importing job file data mentioned above can include process files and task files, and the job objects mentioned above can include process job objects and task job objects.

[0090] It should be noted that the aforementioned process-type files are stored in the `flow` directory of the imported data file package, while task-type files are stored in the `task` directory of the imported data file package. A process-type job can contain multiple task-type jobs. When the job type for imported job file data is a process-type file, the JSON string in the imported job file data can be read, and a JSON conversion tool can be used to convert the JSON string into a Java object of the process job. Then, the task files in the `task` directory are read to determine the associated mapping information. When the job type for imported job file data is a task-type file, the JSON string in the imported job file data can be read, and a JSON conversion tool can be used to convert the JSON string into a Java object of the task job. By processing the mapping information of the process job object or task job object, the mapping information of the local file data can be mapped to the actual information of the local file. The aforementioned JSON string can be a string containing the mapping relationship in the imported job file data.

[0091] In the specific implementation, the backend import module can obtain the import job file data in the import data file package and generate job objects based on the import job file data.

[0092] The backend import module is also used to perform information conversion on the job object according to the data source mapping information, and store the job object into the database of the current environment when the conversion is completed.

[0093] It should be noted that current mapping information can be generated based on the imported mapping information and the current data source. The current mapping information may include data source mapping information, metadata mapping information, resource file mapping information, etc. This embodiment does not impose any restrictions on this.

[0094] Understandably, the import mapping information includes mapping information for the imported file data, such as the data source ID and name corresponding to the imported file. The current mapping information includes mapping information between the new file in the current environment and the imported files in the imported data package, such as the mapping relationship between the name of the new file and the name of the imported file. When the local file is downloaded to the current environment, the mapping information in the local file is not modified, resulting in the mapping information in the local file being identical to that contained in the import mapping information. Based on the current mapping information, the mapping information corresponding to the job object can be mapped and converted, thereby mapping the local file data's mapping information to the actual information of the local file.

[0095] It should be understood that when all the imported files in the imported data file package have been converted, the job object can be stored in the database of the current environment to complete the entire data migration process. The way to store it in the database can be by calling the corresponding DAO operation, and this embodiment does not limit this.

[0096] In the specific implementation, the backend import module performs information transformation on the job object based on the current mapping information, and stores the job object into the database of the current environment to complete the data migration when the information transformation is completed.

[0097] This embodiment obtains import mapping information from the import data file package through the backend import module, and sends the import data source information read from the import mapping information to the frontend module for visualization. It then obtains the current data source selected by the user based on the visualization page, and generates current mapping information based on the current data source and the import mapping information. A job object is generated based on the import job file data obtained from the import data file package. Information transformation is performed on the job object according to the current mapping information, and upon completion of the information transformation, the job object is stored in the current environment's database to complete the data migration. By importing files and replacing the data mapping relationships in the imported files through the backend import module, users only need to upload the import data file package. After the backend import module parses the data and sends the import data source information to the visualization interface of the frontend module, users can select the data source in the current environment to complete the file import and the replacement of the data mapping information in the imported files. The program automatically imports files and replaces mapping information, making deployment fast and flexible, reducing manual steps, and improving the accuracy of imported files.

[0098] Furthermore, to make the generation of mapping information more accurate, the imported data file package also includes: imported resource files; the imported mapping information also includes: imported metadata information;

[0099] It should be noted that the import mapping information may include import data source information and import metadata information; the aforementioned import resource file may be a file that can be used to directly or indirectly implement the functions performed by the aforementioned import job file data.

[0100] It should be explained that the imported resource files mentioned above can include SQL files (Structured Query Language), executable scripts, JAR files (Java Archive), dependency packages, and other files that can be used to help implement the functions of the job file.

[0101] The backend import module is further configured to determine current metadata based on the current data source information and the imported data source information, and generate metadata mapping information based on the imported metadata information and the current metadata; the backend import module is further configured to generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file; correspondingly, the backend import module is further configured to perform an operation of information conversion on the job object based on the data source mapping information, the metadata mapping information and the resource file mapping information.

[0102] It should be noted that when importing files, the import mapping information in the imported data file package can be read. The import mapping information includes the import data source information, import metadata information, and import resource file information. Data source mapping information is generated based on the import data source information in the mapping file and the current data source.

[0103] It is understandable that data source mapping information can represent the mapping relationship between the imported data source and the current data source. The aforementioned data source information may include data source ID, data source name, etc.

[0104] It's important to explain that metadata is the basic unit for managing data on a data platform. It corresponds to tables in a database, topics in a message center, etc. Metadata describes the structure of data, including attributes such as fields, types, and precision. Importing metadata information means importing the metadata information of the imported data. This can be done by querying the current metadata information in the current environment based on the current data source ID and the imported metadata information in the import mapping information, and then generating metadata mapping information based on the current metadata information and the import information.

[0105] It is understandable that metadata mapping information can represent the mapping relationship between imported metadata and current metadata. The aforementioned metadata information may include metadata name, metadata ID, etc.

[0106] The backend import module is further configured to generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file.

[0107] It should be noted that when the backend import module receives the import data file package, it can download the import resource files in the import data file package to the current environment. The aforementioned current resource file is the resource file downloaded from the import data file package to the local machine. Resource file mapping information can be generated based on the import resource file information and the current resource file information.

[0108] It can be explained that the above resource file mapping information can represent the mapping relationship between the current resource file and the imported resource file. The resource file information can include the resource file's code, resource file name, etc.

[0109] In the specific implementation, the backend import module generates metadata mapping information based on the current data source and the imported data source information; determines the current metadata based on the current data source information and the imported data source information; generates metadata mapping information based on the imported metadata information and the current metadata information; generates the current resource file based on the imported resource file; and generates resource file information based on the current resource file and the imported resource file.

[0110] This embodiment generates metadata mapping information based on the current data source and the imported data source information through the backend import module; determines the current metadata based on the current data source information and the imported data source information; generates metadata mapping information based on the imported metadata information and the current metadata information; generates the current resource file based on the imported resource file; generates resource file information based on the current resource file and the imported resource file; and performs mapping and transformation on the mapping information corresponding to the job object based on the data source mapping information, metadata mapping information, and resource file mapping information, so that the mapping information of local file data corresponds to the actual information of the local file, reducing the number of manual operation steps while ensuring the accuracy of the imported information.

[0111] like Figure 2 As shown, Figure 2 This is a flowchart of the core logic processing of the backend import module of the data migration device of the present invention;

[0112] It is understandable that before importing, users need to export files in other environments. The compressed package of exported files is the exported data file package, which is equivalent to the imported data file package in the current environment as described in the above embodiment.

[0113] It should be noted that the backend import module receives and decompresses the uploaded export data file package, which yields mapping files and resource files. The mapping files include metadata information, data source information, and resource file information.

[0114] Furthermore, the backend import module can read the mapping file and return the data source information to the frontend module. The frontend module can then display the data source information in the current environment and the data source information in the mapping file, allowing the user to select a new data source and obtain the mapping information for the new and old data sources.

[0115] Furthermore, the backend import module can query the new and old metadata mapping information based on the new data source ID and metadata name.

[0116] Furthermore, the backend import module can read resource files in the package directory and upload them to obtain the mapping information of new and old resource files.

[0117] Furthermore, the backend import module can read process files in the flow directory and convert them into entity objects, read task files in the task directory, and replace the ID or Code mapping information in the configuration.

[0118] Furthermore, the backend import module calls the DAO operation of the task or process to save the task and process information.

[0119] Furthermore, during the import process, the backend import module will also record the import progress log through the log module and send the import progress log to the frontend module to display the import progress.

[0120] The data migration device of the present invention automatically decompresses the exported data file package and replaces the mapping information through the back-end import module. The entire import process only requires the user to upload and select the data source, which greatly reduces the number of manual operation steps and improves the accuracy and efficiency of import.

[0121] Based on the first embodiment of the data migration device described above, a second embodiment of the data migration device of the present invention is proposed.

[0122] In order to export the job files in the current environment, the device further includes: a backend export module;

[0123] The front-end module is used to visualize the list of exportable jobs when exporting job data, and to send an export request to the back-end export module according to the job selected by the user.

[0124] Understandably, the front-end module can include a visual display interface, an export button, an export confirmation button, and other functional buttons. The front-end module can display a list of exportable files in the current environment through the visual display interface. Users can select the data files to be exported based on the visual interface. When the user has made their selection, the front-end module can send an export request to the back-end module based on the data files selected by the user.

[0125] In practice, the front-end module sends an export request to the back-end module based on the data file selected by the user.

[0126] The backend export module is used to obtain the job information of the data file to be exported according to the export request;

[0127] It should be noted that the above job information may include the export file information of the data file to be exported selected by the user, as well as the dependent file information.

[0128] Furthermore, to facilitate file management, the file types of the data files to be exported include process files and task files; the job information includes first configuration information or second configuration information.

[0129] The backend export module is also used to obtain the first configuration information of the data file to be exported when the file type of the data file to be exported is a task file;

[0130] The backend export module is also used to obtain the process configuration information of the data file to be exported when the file type of the data file to be exported is a process file, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

[0131] It's important to explain that the exported file types include task files. Task files can be files that execute data acquisition tasks or data development tasks. Data acquisition tasks focus on the collection and integration of raw data, corresponding to the collection configuration for one or more tables or files, including the data source and metadata of the collection source, transformation rules, and the data source and metadata of the target. Data development tasks focus on data processing and business development, corresponding to resource files such as SQL, Shell, Python, and Jar packages, supporting offline batch processing and real-time computing.

[0132] It should also be explained that a workflow job can be a series of data acquisition or data development tasks arranged and combined sequentially or in parallel to form a data processing workflow that is closely related in business logic. It supports drag-and-drop visual operations, nested sub-workflows and dependent nodes, and branching processing of conditional nodes. The workflow class file is the file related to completing the workflow job.

[0133] It should be noted that when the backend export module receives an export request, it determines the file type of the data file to be exported and generates different types of export data file packages accordingly. This determination can be based on the attributes of the export file, or it can be generated by the frontend module reading the data file to be exported, or the type selected by the user through the frontend module. This embodiment does not impose any restrictions on this. If the data file to be exported is a task file, the detailed configuration information of the task can be queried based on the task ID or other identification code; this information constitutes the first configuration information. If the data file to be exported is a process file, the detailed configuration information of the process can be queried based on the process ID or other identification code. The detailed information of the task can be recursively queried based on multiple task IDs associated with the process; this information constitutes the second configuration information.

[0134] In the specific implementation, the job information for the data files to be exported is queried by determining the file type of the data files to be exported.

[0135] The backend export module is also used to generate an export mapping file based on the job information;

[0136] It should be noted that the above job information may include data source information, metadata information, and resource file information.

[0137] It is understandable that the exported mapping file is a mapping file generated based on information such as data source information, metadata information, and resource file information. After the exported mapping file is generated, the file can be stored in the mapping directory.

[0138] In the specific implementation, an export mapping file is generated based on the job information that requires the data files to be exported.

[0139] The backend export module is also used to convert the job to be exported and the export mapping file into preset format objects, and store the preset format objects to the corresponding storage directories respectively;

[0140] It should be noted that the preset format object can be a JSON file obtained by converting a JSON file using a JSON conversion tool, or it can be a file of other formats. The format can be converted according to actual needs, and this embodiment does not impose any restrictions on it.

[0141] It should be explained that the aforementioned storage directories can include a flow directory, a task directory, a package directory, and a mapping directory. Process files are converted into preset format objects and named according to their process IDs, then stored in the flow directory; task files are converted into preset format objects and named according to their task IDs, then stored in the task directory; resource files are converted into preset format objects and named according to their file codes, then stored in the package directory; and exported mapping files are stored in the mapping directory. It should be noted that the directory names only represent the file types that the directory should store, and do not limit the names of the storage directories.

[0142] In the actual implementation, the backend export module converts the data files to be exported and the export mapping files into preset format objects, and then stores the preset format objects in the corresponding directory.

[0143] The backend export module is also used to compress the storage directory and generate an export data file package with a preset structure when all the data files to be exported have been converted and stored in the corresponding storage directory.

[0144] It should be noted that the naming convention for exported data file packages can be DIDA_Export_number_type, where the number is auto-incrementing and the type is either task or flow. The following is an example of an exported data file package using JSON as the default format. The default structure of the exported data file package DIDA_Export_0000000001_flow.zip is as follows:

[0145] flow

[0146] --ProcessID.json

[0147] task

[0148] --TaskID.json

[0149] package

[0150] |--File Code

[0151] mapping

[0152] --mapping.json

[0153] In the specific implementation, after all the data files to be exported have been converted and stored in the corresponding directory, the storage directory is compressed and an export data file package with a preset structure is generated.

[0154] In this embodiment, the front-end module initiates an export request to the back-end module based on the user-selected data file to be exported. The back-end export module queries the job information of the data file to be exported by determining its file type; it generates an export mapping file based on the job information; then, it converts the data file to be exported and the export mapping file into a preset format object, and stores the preset format object in the corresponding directory; when all the data files to be exported have been converted and stored in the corresponding directory, the storage directory is compressed and a preset structured export data file package is generated. The user only needs to select the data file to be exported, and the front-end and back-end modules can package the files to be exported. At the same time, corresponding mapping information is generated so that when the export data file package is imported, it can be mapped and replaced according to the mapping information and the configuration information in the import environment, thereby realizing data migration, reducing manual operation steps, and improving the accuracy and efficiency of export.

[0155] like Figure 3 As shown, Figure 3 This is a flowchart of the core logic processing of the backend export module of the data migration device of the present invention;

[0156] Understandably, when the front-end module receives the data file to be exported selected by the user based on the visual interface, it can send an export request to the back-end export module.

[0157] Furthermore, when the backend module receives an export request, it determines the type to be exported based on the `type` parameter. It then queries different configuration information (represented as "task" or "process" in the diagram) by determining whether the data file to be exported is a task or a process.

[0158] Furthermore, when exporting a data file as a task file, query all task information; when exporting a data file as a process file, query all process information, and recursively query task information based on the tasks associated with the process.

[0159] Understandably, the task information includes data source ID, metadata ID, and resource file code. The backend export module queries the data source information based on the data source ID, queries the metadata information based on the metadata ID, and queries and downloads the resource file information based on the resource file code. Then, it organizes mapping information based on the metadata information, data source information, and resource file information, generates an export file based on the mapping information and the data files to be exported, compresses and packages it for download.

[0160] Furthermore, during the export process, the backend export module will also record the export progress log through the log module and send the export progress log to the frontend module to display the export progress.

[0161] In its implementation, the backend export module can receive export requests initiated by the frontend module and determine the type to be exported based on the type parameter of the exported file. It then generates mapping information by querying the file's configuration information and other information, thereby generating the exported file and compressing it. The entire export process only requires the user to select the data file to be exported, and the generation of mapping information and file compression are completed automatically, greatly reducing manual operation steps and improving the accuracy and efficiency of the export.

[0162] Furthermore, to make the import / export progress more intuitive, the device also includes: a log module; the log module is used to generate a running log when the backend export module or the backend import module is running, and send the running log to the frontend module; the frontend module is also used to visualize the execution progress of the backend export module or the backend import module based on the running log.

[0163] Understandably, the logging module can monitor the progress of the backend import and export modules and generate corresponding progress logs to send to the frontend module. The frontend module can then display the export or import progress so that users can intuitively see the export progress.

[0164] It should be noted that the front-end module can also determine the remaining time to complete the export or import task based on the export or import progress, all data files to be exported or imported, and the current time elapsed, and display the remaining time on the visualization interface.

[0165] This embodiment generates runtime logs for the backend import or export module through the log module and sends the runtime logs to the frontend module, thereby enabling a visual display of the export / import progress.

[0166] Reference Figure 4 , Figure 4 This is a schematic diagram of an application scenario of the data migration device of the present invention;

[0167] like Figure 4 As shown, data can be exported using a data migration device installed in environment A and imported using a data migration device installed in environment B. The overall export and import process will be explained below.

[0168] Understandably, at the start of the export process, the user selects the person or task and clicks export. The data file package is then exported by querying the task process information through the backend export module. For details of the export process, please refer to the export module (i.e., the backend export module described in the above embodiment), which will not be elaborated here.

[0169] Furthermore, users can upload the exported data file package generated in environment A to environment B. After clicking import, the exported data file package parsing mapping information can be uploaded. For details of the import process, please refer to the import module (i.e., the backend import module described in the above embodiment), which will not be elaborated here.

[0170] Furthermore, after parsing the mapping information, the data source information is sent to the front-end module for display. Users can select a new data source based on the front-end module. The back-end import module modifies the mapping information and saves the task flow information by modifying the data of the new data source and the imported data file package (see import module for details). When the task information or flow information is uploaded to the database, it indicates that the file import is complete and the import ends.

[0171] With the data migration device, users only need "one interaction and two clicks" to complete the data migration. One interaction refers to selecting and confirming the data source only once during import; two clicks refer to clicking a button once during export and once during import. Regardless of the number of tasks and processes, complex and tedious configuration modifications and the need for manual uploading and associating of resource files can be transformed into fully automated processes through simple operations on the page. This allows for flexible and rapid migration and deployment of task flow information in any environment, greatly reducing manual operations and improving the accuracy and convenience of data migration.

[0172] This invention also provides a data migration method, referring to... Figure 5 Figure 5 Figure 5 This is a flowchart illustrating the first embodiment of the data migration method of the present invention.

[0173] In this embodiment, the data migration method includes the following steps:

[0174] S10: Obtain the import mapping information from the import data file package, and read the import data source information from the import mapping information;

[0175] S20: Query all data sources in the current environment, and send the queried data source information and the imported data source information to the front-end module for visualization;

[0176] It should be noted that the executing entity of the method in this embodiment can be a terminal device with common functions of data processing and interface display, such as a personal computer, or an electronic device with the same or similar functions, such as a device equipped with the data migration device described above, referred to as a data migration device. The following uses a data migration device as an example to describe this embodiment and the following embodiments.

[0177] It is understandable that data migration requires exporting and importing the data to be migrated. The data to be migrated is exported from the old environment to generate an import data file package that can be imported into the new environment. It is easy to understand that the import data file package can include all or part of the information in the original environment, and the imported files can be files selected by the user in the old environment according to their needs. This embodiment does not impose any restrictions on this.

[0178] It should be noted that the aforementioned import data file package may include import resource files and import mapping information. The import mapping information is stored in the mapping directory of the import data file package and contains mapping information of the data in the import resource files. When the backend import module obtains this mapping information, it can perform mapping replacement or generate new mapping information based on the mapping information.

[0179] It should be noted that the aforementioned imported resource files are stored in the package directory of the imported data file package, and may include dependency files, script files, etc. This embodiment does not impose any restrictions on this.

[0180] It should be noted that the data migration device can display information about the imported data source and the data sources currently existing in the environment, allowing users to select the data source they need.

[0181] In practice, the data migration device can obtain the import mapping information in the imported data file package and visualize the imported data source information read from the import mapping information.

[0182] S30: Obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source and the current data source;

[0183] It is understandable that the above current data source is the data source in the current environment selected by the user based on the imported data source information.

[0184] It should be noted that the above data source mapping information is determined based on the imported data source information and the current data source.

[0185] In the specific implementation, the data migration process determines the current data source selected by the user based on the visual display interface, and generates data source mapping information based on the imported data source and the current data source.

[0186] S40: Obtain the import job file data from the import data file package, and generate a job object based on the import job;

[0187] Understandably, the data migration device can download the files in the import data file package to the current environment to generate local files. It can also obtain the import job file data in the import data file package and generate job objects based on the import job file data. The attributes of the aforementioned job objects include data sources, metadata, and data mappings of resource files.

[0188] It should be explained that the job types for importing job file data mentioned above can include process files and task files, and the job objects mentioned above can include process job objects and task job objects.

[0189] It should be noted that the aforementioned process-type files are stored in the `flow` directory of the imported data file package, while task-type files are stored in the `task` directory of the imported data file package. A process-type job can contain multiple task-type jobs. When the job type for imported job file data is a process-type file, the JSON string in the imported job file data can be read, and a JSON conversion tool can be used to convert the JSON string into a Java object of the process job. Then, the task files in the `task` directory are read to determine the associated mapping information. When the job type for imported job file data is a task-type file, the JSON string in the imported job file data can be read, and a JSON conversion tool can be used to convert the JSON string into a Java object of the task job. By processing the mapping information of the process job object or task job object, the mapping information of the local file data can be mapped to the actual information of the local file. The aforementioned JSON string can be a string containing the mapping relationship in the imported job file data.

[0190] In practice, the data migration device can generate job objects based on the import job file data obtained from the import data file package.

[0191] S50: Perform information transformation on the job object according to the data source mapping information, and store the job object in the database of the current environment when the transformation is completed;

[0192] It should be noted that current mapping information can be generated based on the imported mapping information and the current data source. The current mapping information may include data source mapping information, metadata mapping information, resource file mapping information, etc. This embodiment does not impose any restrictions on this.

[0193] Understandably, the import mapping information includes mapping information for the imported file data, such as the data source ID and name corresponding to the imported file. The current mapping information includes mapping information between the new file in the current environment and the imported files in the imported data package, such as the mapping relationship between the name of the new file and the name of the imported file. When the local file is downloaded to the current environment, the mapping information in the local file is not modified, resulting in the mapping information in the local file being identical to that contained in the import mapping information. Based on the current mapping information, the mapping information corresponding to the job object can be mapped and converted, thereby mapping the local file data's mapping information to the actual information of the local file.

[0194] It should be understood that when all the imported files in the imported data file package have been converted, the job object can be stored in the database of the current environment to complete the entire data migration process. The way to store it in the database can be by calling the corresponding DAO operation, and this embodiment does not limit this.

[0195] In practice, the data migration device performs information conversion on the job object based on the current mapping information, and stores the job object into the database of the current environment when the information conversion is completed to complete the data migration.

[0196] In this embodiment, the migration device acquires import mapping information from the import data file package and visualizes the import data source information read from the import mapping information. It then acquires the current data source selected by the user based on the visualization page and generates current mapping information based on the current data source and the import mapping information. A job object is generated based on the import job file data acquired from the import data file package. Information transformation is performed on the job object according to the current mapping information, and upon completion of the information transformation, the job object is stored in the current environment's database to complete the data migration. By mapping and replacing the data mapping relationships in the file, the user only needs to upload the import data file package and select the current data source to complete the file import and the replacement of the data mapping information in the imported file. The program automatically performs the file import and mapping information replacement, resulting in fast and flexible deployment, reducing manual operation steps, and improving the accuracy of imported files.

[0197] Furthermore, to make the generation of mapping information more accurate, the imported data file package also includes: imported resource files; the imported mapping information also includes: imported metadata information;

[0198] It should be noted that the import mapping information may include import data source information and import metadata information; the aforementioned import resource file may be a file that can be used to directly or indirectly implement the functions performed by the aforementioned import job file data.

[0199] It should be explained that the imported resource files mentioned above can include SQL files (Structured Query Language), execution scripts, JAR files, dependency packages, and other files that can be used to help achieve the functions implemented by the job file.

[0200] After the steps of obtaining the current data source selected by the user based on the visual display page and generating data source mapping information based on the imported data source information and the current data source, the method further includes: determining the current metadata based on the current data source information and the imported data source information, and generating metadata mapping information based on the imported metadata information and the current metadata; generating the current resource file based on the imported resource file, and generating resource file mapping information based on the current resource file and the imported resource file; correspondingly, the step of performing information transformation on the job object based on the data source mapping information further includes: performing information transformation on the job object based on the data source mapping information, the metadata mapping information, and the resource file mapping information.

[0201] It should be noted that the import mapping information may include import data source information and import metadata information; the aforementioned import resource file may be a file that can be used to directly or indirectly implement the functions performed by the aforementioned import job file data.

[0202] It should be explained that the imported resource files mentioned above can include SQL files (Structured Query Language), execution scripts, JAR files, dependency packages, and other files that can be used to help achieve the functions implemented by the job file.

[0203] It should be noted that when importing files, the import mapping information in the imported data file package can be read. The import mapping information includes the import data source information, import metadata information, and import resource file information. Data source mapping information is generated based on the import data source information in the mapping file and the current data source.

[0204] It is understandable that data source mapping information can represent the mapping relationship between the imported data source and the current data source. The aforementioned data source information may include data source ID, data source name, etc.

[0205] It's important to explain that metadata is the basic unit for managing data on a data platform. It corresponds to tables in a database, topics in a message center, etc. Metadata describes the structure of data, including attributes such as fields, types, and precision. Current metadata information can be queried in the current environment based on the current data source ID and the import metadata information in the import mapping information. Metadata mapping information can then be generated based on the current metadata information and the import information.

[0206] It is understandable that metadata mapping information can represent the mapping relationship between imported metadata and current metadata. The aforementioned metadata information may include metadata name, metadata ID, etc.

[0207] It should be noted that when the data migration device receives the import data file package, it can download the import resource files in the import data file package to the current environment. The aforementioned current resource file is the resource file downloaded from the import data file package to the local machine. Resource file mapping information can be generated based on the import resource file information and the current resource file information.

[0208] It can be explained that the above resource file mapping information can represent the mapping relationship between the current resource file and the imported resource file. The resource file information can include the resource file's code, resource file name, etc.

[0209] In the specific implementation, the data migration device generates metadata mapping information based on the current data source and the imported data source information; determines the current metadata based on the current data source information and the imported data source information; generates metadata mapping information based on the imported metadata information and the current metadata information; generates the current resource file based on the imported resource file; and generates resource file information based on the current resource file and the imported resource file.

[0210] In this embodiment, the data migration device generates metadata mapping information based on the current data source and the imported data source information; determines the current metadata based on the current data source and the imported data source information; generates metadata mapping information based on the imported metadata information and the current metadata information; generates the current resource file based on the imported resource file; generates resource file information based on the current resource file and the imported resource file; and performs mapping and transformation on the mapping information corresponding to the job object based on the data source mapping information, metadata mapping information, and resource file mapping information, so that the mapping information of the local file data corresponds to the actual information of the local file, reducing the number of manual operation steps while ensuring the accuracy of the imported information.

[0211] Based on the first embodiment of the data migration method described above, a second embodiment of the data migration method of the present invention is proposed, referring to... Figure 6 , Figure 6 This is a flowchart illustrating a second embodiment of the data migration method of the present invention.

[0212] To export the job files from the current environment, the method further includes:

[0213] S100: When exporting job data, the list of exportable jobs is displayed visually, and an export request is sent to the backend export module according to the data file to be exported selected by the user.

[0214] S200: Obtain the job information of the data file to be exported according to the export request;

[0215] Understandably, users can select the data files to be exported from the list of exportable files displayed on the data migration device's visual interface, and obtain the job information of the data files to be exported when the user completes the selection.

[0216] It should be noted that the above job information refers to the export file information and dependent file information of the data files to be exported selected by the user.

[0217] To facilitate file management, the file types of the data files to be exported include process files and task files; the job information includes first configuration information or second configuration information; the step of obtaining the job information of the data files to be exported based on the user-selected data files includes:

[0218] S201: When the file type of the data file to be exported is a task file, obtain the first configuration information of the data file to be exported;

[0219] S202: When the file type of the data file to be exported is a process file, obtain the process configuration information of the data file to be exported, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

[0220] It should be noted that the data migration device can determine the file type of the data file to be exported and generate different export data file packages. If the data file to be exported is a task file, the detailed configuration information of the task can be queried according to the task ID or other identification code. This information is the first configuration information. If the data file to be exported is a process file, the detailed configuration information of the process can be queried according to the process ID or other identification code. The detailed information of the task can be recursively queried according to the multiple task IDs associated with the process. This information is the second configuration information.

[0221] In the specific implementation, the job information for the data files to be exported is queried by determining the file type of the data files to be exported.

[0222] S300: Generate an export mapping file based on the job information;

[0223] It should be noted that the above job information may include data source information, metadata information, and resource file information.

[0224] It is understandable that the exported mapping file is a mapping file generated based on information such as data source information, metadata information, and resource file information, and is stored in the mapping directory.

[0225] In the specific implementation, an export mapping file is generated based on the job information that requires the data files to be exported.

[0226] S400: Convert the data file to be exported and the export mapping file into preset format objects, and store the preset format objects into their respective storage directories;

[0227] It should be noted that the preset format object can be a JSON file obtained by converting a JSON file using a JSON conversion tool, or it can be a file of other formats. The format can be converted according to actual needs, and this embodiment does not impose any restrictions on it.

[0228] It should be explained that the aforementioned storage directories can include a flow directory, a task directory, a package directory, and a mapping directory. Process files are converted into preset format objects and named according to their process IDs, then stored in the flow directory; task files are converted into preset format objects and named according to their task IDs, then stored in the task directory; resource files are converted into preset format objects and named according to their file codes, then stored in the package directory; and exported mapping files are stored in the mapping directory. It should be noted that the directory names only represent the file types that the directory should store, and do not limit the names of the storage directories.

[0229] In practice, the data migration device converts the data files to be exported and the export mapping files into preset format objects, and then stores the preset format objects in the corresponding directory.

[0230] S500: When all the data files to be exported have been converted and stored in the corresponding storage directory, the storage directory is compressed to generate an export data file package with a preset structure.

[0231] It should be noted that the naming convention for exported data file packages can be DIDA_Export_number_type, where the number is auto-incrementing and the type is either task or flow. The following is an example of an exported data file package using JSON as the default format. The default structure of the exported data file package DIDA_Export_0000000001_flow.zip is as follows:

[0232] flow

[0233] |--Process ID.json

[0234] task

[0235] --TaskID.json

[0236] package

[0237] |--File Code

[0238] mapping

[0239] --mapping.json

[0240] In the specific implementation, after all the data files to be exported have been converted and stored in the corresponding directory, the storage directory is compressed and an export data file package with a preset structure is generated.

[0241] In this embodiment, the data migration device determines the file type of the data file to be exported and queries the job information of the data file to be exported; then, it generates an export mapping file based on the job information of the data file to be exported; next, it converts the data file to be exported and the export mapping file into a preset format object, and then stores the preset format object in the corresponding directory; when all the data files to be exported have been converted and stored in the corresponding directory, the storage directory of the preset format object is compressed to generate an export data file package with a preset structure. Users only need to select the data files to be exported, and the system will automatically package the files to be exported, while generating corresponding mapping information so that when the export data file package is imported, it can be mapped and replaced according to the mapping information and the configuration information in the import environment, thereby realizing data migration, reducing manual operation steps, and improving the accuracy and efficiency of export.

[0242] Furthermore, to make the import / export progress more intuitive, the method also includes: generating a running log when importing or exporting data files; and visually displaying the running log.

[0243] Understandably, when performing import or export operations, corresponding operation logs can be generated, and the execution progress can be displayed through a visual interface, allowing users to more intuitively see the progress of data migration.

[0244] In this embodiment, the data migration device can generate an operation log based on the recorded data migration progress. The execution status of the data migration can be visualized through a visual display interface based on the operation log, allowing users to obtain the data migration completion progress more intuitively.

[0245] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0246] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0247] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0248] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data migration device, characterized in that, The device includes: a backend import module and a frontend module; The backend import module is used to obtain import mapping information from the import data file package and read import data source information from the import mapping information; The backend import module is also used to query all data sources in the current environment and send the queried data source information and the imported data source information to the frontend module for visualization. The backend import module is also used to obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source information and the current data source; The backend import module is also used to obtain import job file data in the import data file package and generate job objects based on the import job; The backend import module is also used to perform information conversion on the job object according to the data source mapping information, and store the job object into the database of the current environment when the conversion is completed.

2. The data migration apparatus as described in claim 1, characterized in that, The device further includes: a backend export module; The front-end module is used to visualize the list of exportable jobs when exporting job data, and to send an export request to the back-end export module according to the job selected by the user. The backend export module is used to obtain job information of the data file to be exported based on the export request; The backend export module is also used to generate an export mapping file based on the job information; The backend export module is also used to convert the job to be exported and the export mapping file into preset format objects, and store the preset format objects to the corresponding storage directories respectively; The backend export module is also used to compress the storage directory and generate an export data file package with a preset structure when all the data files to be exported have been converted and stored in the corresponding storage directory.

3. The data migration apparatus as described in claim 1, characterized in that, The imported data file package also includes: imported resource files; The import mapping information also includes: import metadata information; The backend import module is further configured to determine the current metadata based on the current data source information and the import data source information, and generate metadata mapping information based on the import metadata information and the current metadata; The backend import module is further configured to generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file; Correspondingly, the backend import module is also used to perform information conversion operations on the job object based on the data source mapping information, the metadata mapping information, and the resource file mapping information.

4. The data migration apparatus as described in claim 2, characterized in that, The file types of the data files to be exported include process files and task files; The job information includes either first configuration information or second configuration information; The backend export module is also used to obtain the first configuration information of the data file to be exported when the file type of the data file to be exported is a task file; The backend export module is also used to obtain the process configuration information of the data file to be exported when the file type of the data file to be exported is a process file, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

5. The data migration apparatus as described in claim 2, characterized in that, The device further includes: a log module; The log module is used to generate runtime logs when the backend export module or the backend import module is running, and to send the runtime logs to the frontend module. The front-end module is also used to visualize the execution progress of the back-end export module or the back-end import module based on the running log.

6. A data migration method, characterized in that, The method is applied to the data migration apparatus as described in any one of claims 1-5, and the method includes: Obtain the import mapping information from the imported data file package, and read the import data source information from the import mapping information; The system retrieves all data sources in the current environment and provides a visual display of the retrieved data source information and the imported data source information. Obtain the current data source selected by the user based on the visual display page, and generate data source mapping information based on the imported data source information and the current data source; Obtain the import job file data from the import data file package, and generate a job object based on the import job; The job object is transformed according to the data source mapping information, and the job object is stored in the database of the current environment when the transformation is completed.

7. The method as described in claim 6, characterized in that, The method further includes: When exporting job data, the list of exportable jobs is displayed visually, and an export request is sent to the backend export module based on the job selected by the user. Obtain job information for the data files to be exported based on the export request; Generate an export mapping file based on the job information; The data file to be exported and the export mapping file are converted into preset format objects, and the preset format objects are stored in their respective storage directories. Once all the data files to be exported have been converted and stored in the corresponding storage directory, the storage directory is compressed to generate an export data file package with a preset structure.

8. The method as described in claim 6, characterized in that, The imported data file package also includes: imported resource files; The import mapping information also includes: import metadata information; After the steps of obtaining the current data source selected by the user based on the visual display page, and generating data source mapping information based on the imported data source information and the current data source, the method further includes: The current metadata is determined based on the current data source information and the imported data source information, and metadata mapping information is generated based on the imported metadata information and the current metadata; Generate a current resource file based on the imported resource file, and generate resource file mapping information based on the current resource file and the imported resource file; Accordingly, the step of converting the information of the job object based on the data source mapping information further includes: The job object is transformed based on the data source mapping information, the metadata mapping information, and the resource file mapping information.

9. The method as described in claim 7, characterized in that, The file types of the data files to be exported include process files and task files; The job information includes either first configuration information or second configuration information; The step of obtaining the job information of the data file to be exported based on the user-selected data file includes: When the file type of the data file to be exported is a task file, obtain the first configuration information of the data file to be exported; When the file type of the data file to be exported is a process file, obtain the process configuration information of the data file to be exported, and recursively query the second configuration information of the task to which the data file to be exported depends based on the process configuration information.

10. The method as described in claim 7, characterized in that, The method further includes: When importing or exporting data files, a runtime log is generated; The operation logs are visualized.

Images