Data migration method and device based on task scheduling, electronic equipment and medium

Abstract

The invention discloses a data migration method and device based on task scheduling, electronic equipment and a medium, and can be widely applied to the technical field of data processing. The method is applied to a distributed task scheduling end, after a real-time migration task of a data emigration end and to-be-migrated data corresponding to the real-time migration task are obtained in the distributed task scheduling end, a migration scheduling strategy is configured according to the to-be-migrated data, and then the to-be-migrated data is preprocessed to obtain a preprocessing result; and executing a real-time migration task according to the preprocessing result and the migration scheduling strategy to transfer the to-be-migrated data from the data emigration end to the data immigration end, so that a distributed data migration function can be realized, and the data migration efficiency and the migration accuracy are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method and device based on task scheduling, an electronic device and a medium. Background Art

[0002] In related technologies, with the acceleration of urbanization and economic restructuring, the original urban division boundaries can no longer meet actual needs. In order to better adapt to the development of urbanization, optimize administrative efficiency, coordinate urban and rural development, etc., it is necessary to reasonably adjust the existing urban division domains. The adjustment of urban division domains will involve a large amount of data migration operations. At present, it mainly relies on the corresponding system in each urban division domain to set the migration method and then migrate data separately. This migration method has problems such as low migration efficiency and accuracy.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data migration method and device, electronic device and medium based on task scheduling, which can effectively improve data migration efficiency and migration accuracy.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application proposes a data migration method based on task scheduling in one aspect. The method is applied to a distributed task scheduling end, and the distributed task scheduling end interacts with multiple data ends, and the data ends include a data migration end and a data migration end. The method includes the following steps:

[0006] Acquire the real-time migration task of the data migration-out end and the to-be-migrated data corresponding to the real-time migration task;

[0007] Configure a migration scheduling strategy according to the data to be migrated;

[0008] Preprocessing the data to be migrated to obtain a preprocessing result;

[0009] The real-time migration task is executed according to the preprocessing result and the migration scheduling strategy to transfer the data to be migrated from the data migration-out end to the data migration-in end.

[0010] In some embodiments, configuring a migration scheduling strategy according to the data to be migrated includes:

[0011] Obtaining business scenario information corresponding to the data to be migrated;

[0012] The data migration timing information in the migration scheduling strategy is configured according to the business scenario information.

[0013] In some embodiments, configuring a migration scheduling strategy according to the data to be migrated includes:

[0014] Acquire the priority of the data to be migrated and the dependency relationship between the plurality of data to be migrated;

[0015] The data migration order in the migration scheduling policy is configured according to the priority and the dependency relationship.

[0016] In some embodiments, configuring a migration scheduling strategy according to the data to be migrated includes:

[0017] Calculating the amount of data to be migrated;

[0018] The data migration mode in the migration scheduling strategy is configured according to the data volume.

[0019] In some embodiments, configuring a migration scheduling strategy according to the data to be migrated includes:

[0020] Acquire lifecycle information of the data to be migrated and business requirement information corresponding to the data to be migrated;

[0021] Analyzing the usage frequency of the data to be migrated;

[0022] determining a correspondence relationship of the data to be migrated according to the life cycle information and the usage frequency, wherein the correspondence relationship is used to characterize a mapping relationship between a data partition and a data type of the data to be migrated;

[0023] The resource allocation information in the migration scheduling strategy is configured according to the corresponding relationship and the business requirement information.

[0024] In some embodiments, preprocessing the data to be migrated to obtain a preprocessing result includes:

[0025] Cleaning the data to be migrated to obtain first migration data;

[0026] performing format conversion on the first migration data to obtain second migration data;

[0027] The data verification identifier in the second migration data is verified to obtain the preprocessing result.

[0028] In some embodiments, when executing the real-time migration task according to the preprocessing result and the migration scheduling strategy, the method further includes the following steps:

[0029] Obtaining data migration report information corresponding to the real-time migration task;

[0030] Generate migration efficiency and resource usage status information according to the data migration report information;

[0031] The migration scheduling strategy is optimized according to the migration efficiency and the resource usage status information.

[0032] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a data migration device based on task scheduling, wherein the device is arranged at a distributed task scheduling end, and the distributed task scheduling end interacts with a plurality of data ends, wherein the data ends include a data migration end and a data migration end, and the device includes:

[0033] The first module is used to obtain the real-time migration task of the data migration-out end and the data to be migrated corresponding to the real-time migration task;

[0034] The second module is used to configure a migration scheduling strategy according to the data to be migrated;

[0035] The third module is used to preprocess the data to be migrated to obtain a preprocessing result;

[0036] The fourth module is used to execute the real-time migration task according to the preprocessing result and the migration scheduling strategy to transfer the data to be migrated from the data migration-out end to the data migration-in end.

[0037] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:

[0038] at least one processor;

[0039] at least one memory for storing at least one program;

[0040] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0041] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0042] The embodiments of the present application include at least the following beneficial effects: The present application provides a data migration method and device based on task scheduling, an electronic device and a medium. The scheme is applied to a distributed task scheduling end to obtain the real-time migration task of the data migration end and the data to be migrated corresponding to the real-time migration task in the distributed task scheduling end, configure the migration scheduling strategy according to the data to be migrated, and then pre-process the data to be migrated to obtain the pre-processing result. Then, the real-time migration task is executed according to the pre-processing result and the migration scheduling strategy to transfer the data to be migrated from the data migration end to the data migration end, thereby realizing the distributed data migration function and effectively improving the migration accuracy of the data migration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a data migration method based on task scheduling provided in an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of the application interaction architecture of the data migration method based on task scheduling provided in an embodiment of the present application;

[0045] Figure 3 This is a first seed flow chart of step S120 provided in an embodiment of the present application;

[0046] Figure 4 is a second seed flow chart of step S120 provided in an embodiment of the present application;

[0047] Figure 5 is a third seed flow chart of step S120 provided in an embodiment of the present application;

[0048] Figure 6 is a fourth sub-flowchart of step S120 provided in an embodiment of the present application;

[0049] Figure 7 is a flowchart of preprocessing data to be migrated provided by an embodiment of the present application;

[0050] Figure 8 is a complete implementation flow chart of the data migration method based on task scheduling provided in an embodiment of the present application;

[0051] Fig. 9 is a structural diagram of a data migration device based on task scheduling provided in an embodiment of the present application;

[0052] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0054] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0055] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0058] XXL-JOB (also known as XX-JOB or XXL) is a lightweight, easily scalable distributed task scheduling platform. The platform is designed to solve the problem of scheduled tasks in a distributed environment, provides rich functional features, and is widely used in various enterprises and projects.

[0059] Data migration is the process of transferring data from one storage system to another, which usually includes selecting, preparing, extracting and transforming data so that the data can be permanently transferred from one computer system to another. The implementation of data migration requires careful planning and preparation, including understanding the data source, establishing a data dictionary, analyzing data structure and code differences, developing ETL tools, etc. Data migration is essential to maintain the availability and integrity of data, while also improving storage efficiency.

[0060] Task scheduling is a core function in the operating system. It allocates and manages multiple tasks in the computer according to certain strategies and algorithms to optimize the utilization of system resources and improve the efficiency of task execution. Task scheduling is a key technology to ensure the efficient operation of computer systems. Through complex algorithms and strategies, tasks are reasonably allocated and managed to maximize the performance and response speed of the system.

[0061] In related technologies, with the acceleration of urbanization and economic restructuring, the original urban division boundaries can no longer meet actual needs. In order to better adapt to the development of urbanization, optimize administrative efficiency, and coordinate urban and rural development, it is necessary to reasonably adjust the existing urban division domains. When the urban division domain is adjusted, in order to ensure the accuracy and timeliness of the data, the corresponding system data of the urban division domain needs to be migrated synchronously. The current migration method mainly relies on the corresponding system in each urban division domain to set up its own migration method to migrate data separately, resulting in low migration efficiency and migration accuracy.

[0062] In view of this, a data migration method and device, electronic device and medium based on task scheduling are provided in the embodiments of the present application. The present application interacts with multiple data migration ends and data migration ends respectively through a distributed task scheduling end, and obtains the real-time migration task of the data migration end and the data to be migrated corresponding to the real-time migration task in the distributed task scheduling end, configures the migration scheduling strategy according to the data to be migrated, and then pre-processes the data to be migrated to obtain the pre-processing result, and executes the real-time migration task according to the pre-processing result and the migration scheduling strategy to transfer the data to be migrated from the data migration end to the data migration end, thereby realizing the distributed data migration function and effectively improving the migration accuracy of the data migration efficiency.

[0063] The data migration method based on task scheduling provided in the embodiment of the present application relates to the field of data processing technology. The data migration method based on task scheduling provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a data migration method based on task scheduling, etc., but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] Figure 1 is an optional flowchart of a data migration method based on task scheduling provided in an embodiment of the present application. Figure 1 The method in can be applied to Figure 2 In the interactive architecture diagram shown. Specifically, Figure 2 The interactive architecture shown includes a distributed task scheduling terminal 210 and multiple data terminals. The distributed task scheduling terminal can adopt but is not limited to XXL-JOB. The multiple data terminals include at least one data migration terminal 220 and at least one data migration terminal 230. It is understandable that the distributed task scheduling terminal can provide a web interface, so that the target object can create, edit, delete migration tasks, and view the execution log and execution status of the data migration task.

[0066] In the embodiments of the present application, Figure 1 The method may include but is not limited to steps S110 to S120:

[0067] Step S110, obtaining the real-time migration task of the data migration-out end and the to-be-migrated data corresponding to the real-time migration task;

[0068] It is understandable that the real-time migration task can be created by the target object in the human-computer interaction interface provided by the distributed task scheduling end. The creation process requires the target object to fill in the name of the real-time migration task, the source location and the target location of the migration data corresponding to the real-time migration task. For example, when it is necessary to migrate the data of Project 1 of the data out-migration end A1 to the data in-migration end B1, the target object can enter the name of the real-time migration task as "Project 1", the source location as the data out-migration end A1, and the target location as the data in-migration end B1 in the corresponding position of the human-computer interaction interface provided by the distributed task scheduling end. After that, the distributed task scheduling end can automatically generate a real-time migration task, and the data of the real-time migration task is all the data involved in Project 1.

[0069] Step S120: configuring a migration scheduling strategy according to the data to be migrated;

[0070] It is understandable that the real-time migration task is created in real time, and the data migration end and migration task environment involved in each real-time migration task are different. Therefore, different migration scheduling strategies are required to carry out the migration process of the data to be migrated in order to effectively improve the migration efficiency of the data to be migrated. This embodiment can effectively improve the efficiency of data migration by configuring the migration scheduling strategy according to the data to be migrated. Specifically, the configuration process of this embodiment is automatically configured at the distributed task scheduling end, so that the target object does not need to worry about the specific scheduling logic, but only needs to focus on realizing its own unique migration business logic, so that the data migration work can be successfully completed. Exemplarily, when the amount of data to be migrated is large, the migration scheduling strategy can be configured to migrate the data to the node with large bandwidth and idleness. At the same time, the migration strategy can also be configured to split the data to be migrated into multiple small tasks for parallel execution, thereby effectively improving the efficiency of data migration.

[0071] Step S130, preprocessing the data to be migrated to obtain a preprocessing result;

[0072] It is understandable that the data formats corresponding to the two migration ends involved in the data to be migrated may be different. Therefore, in this embodiment, the data to be migrated can be pre-processed before data migration. For example, when the data format of the data to be migrated at the data migration end is JSON and the data migration end is YAML, the data format of the data to be migrated can be pre-processed into YAML before data migration, so that the data migration end can accurately and effectively receive the migrated data.

[0073] Step S140: Execute the real-time migration task according to the preprocessing result and the migration scheduling strategy to transfer the data to be migrated from the data migration-out end to the data migration-in end.

[0074] It is understandable that after obtaining the preprocessing results of the data to be migrated and the migration scheduling strategy, this embodiment automatically calls the resource information of the migration process through the migration scheduling strategy to ensure that the data to be migrated is accurately transferred from the data migration end to the data migration end. Exemplarily, when the migration scheduling strategy is "calling multiple currently idle nodes and splitting the data to be migrated into multiple small tasks", before data migration, the distributed task scheduling end calls nodes 1, 2, and 3 that are currently idle, and splits the data to be migrated into 3 small tasks according to the bandwidth status of nodes 1, 2, and 3, and then controls nodes 1, 2, and 3 to execute the corresponding small tasks in parallel to complete the transmission process of the data to be migrated in parallel.

[0075] It can be seen that this embodiment uses a distributed task scheduling terminal to perform data migration functions, so that scheduling tasks can be executed in a distributed environment to ensure that data migration tasks are executed according to preset requirements. In addition, the distributed task scheduling terminal also provides a Web interface to facilitate the creation, editing, and deletion of migration tasks for target objects, as well as viewing migration task execution logs and monitoring task status. It also supports dynamic addition, modification, and deletion of tasks at runtime without stopping the entire application. In addition, by configuring the migration scheduling strategy, the embodiment of the present application allows data migration personnel to successfully complete data migration without having to worry about specific scheduling logic, and only need to focus on implementing their own unique migration business logic. This embodiment uses the distributed task scheduling end to configure the data migration scheduling strategy, so that when there are new functional requirements in the subsequent stage, the data migration personnel can easily create migration tasks after completing their own migration logic development work, thereby realizing a fully automated migration process. For example, assuming that a migration function for data in a specific format needs to be added in the future, the data migration personnel only need to implement the public abstract class according to the established specifications and define the data reading, conversion and writing logic, and they can automatically include it in the migration task sequence without modifying the overall scheduling framework, which greatly improves the development efficiency and the flexibility of data migration; for example, when the existing migration logic needs to be optimized or repaired, since each migration task logic is relatively independent, the data migration personnel can directly modify the specific migration business logic without affecting other unrelated migration task logics, reducing the complexity of maintenance and improving the scalability and maintainability of data migration by the distributed task scheduling end.

[0076] In the embodiments of the present application, Figure 3 As shown, the process of configuring the migration scheduling strategy according to the data to be migrated includes but is not limited to step S310 and step S320:

[0077] Step S310: Obtain business scenario information corresponding to the data to be migrated;

[0078] Step S320: configuring data migration timing information in the migration scheduling strategy according to the business scenario information.

[0079] It is understandable that the business scenario information includes but is not limited to the business volume, the load condition of the system where the business is located, business demand information, etc. Taking a large business volume as an example, since the data to be migrated corresponding to the business volume needs to occupy a large bandwidth, if data migration is performed during the peak period of the business, it will affect the stability of the overall business execution. Therefore, this embodiment can configure the data migration timing information in the migration scheduling strategy according to the business scenario information. Specifically, the data migration timing information can be set to perform a large-scale migration process of the data to be migrated during the off-peak period of business execution, thereby effectively reducing the impact on the normal business operation process. Taking the load condition of the system where the business is located as multiple loads running simultaneously as an example, the data migration timing information in the migration scheduling strategy can be configured according to the load condition to perform data migration only when a small amount of load is running, so that the subsequent data migration task will only be executed when a small amount of load is running, thereby reducing the impact on the load operation process and improving the efficiency of data transmission.

[0080] In the embodiments of the present application, Figure 4 As shown, the process of configuring the migration scheduling strategy according to the data to be migrated includes but is not limited to step S410 and step S420:

[0081] Step S410: obtaining the priority of the data to be migrated and the dependency relationship between the multiple data to be migrated;

[0082] Step S420: configure the data migration order in the migration scheduling strategy according to the priority and dependency relationship.

[0083] It can be understood that the priority of the data to be migrated can be a level set by the target object according to the role of the data to be migrated in the business operation. Exemplarily, the current data to be migrated includes data 1 to be migrated, data 2 to be migrated, and data 3 to be migrated. Data 1 to be migrated can affect the smoothness of the corresponding business operation, data 2 to be migrated can affect whether the corresponding business is started, and data 3 to be migrated is an insignificant parameter generated during the operation of the corresponding business. Therefore, the migration level of data 2 to be migrated can be set to the first priority, the migration level of data 1 to be migrated can be set to the second priority, and the migration level of data 3 to be migrated can be set to the third priority. After setting the priority of the corresponding data to be migrated, this embodiment can configure the data migration order in the migration scheduling policy based on the priority, so that the migration process of the data to be migrated will not affect the normal operation of the corresponding business.

[0084] It can be understood that the dependency relationship of the data to be migrated can be a mutual influence relationship set by the target object according to the degree of mutual influence of the data to be migrated in the business operation. Exemplarily, the current data to be migrated includes data 1 to be migrated, data 2 to be migrated, and data 3 to be migrated. For business 1, data 2 to be migrated is data that affects whether business 1 runs, data 1 to be migrated is data that affects the stability of business 1 during operation, and data 3 to be migrated is a parameter that business 1 will use as long as it runs. It can be seen that in order for business 1 to run normally at the data migration end, data 2 to be migrated and data 3 to be migrated must exist at the same time. Therefore, data 2 to be migrated and data 3 to be migrated are mutually influential data. Therefore, this embodiment pre-sets the dependency relationship between data 2 to be migrated and data 3 to be migrated, and configures the data migration order in the migration scheduling strategy according to the dependency relationship, so that the corresponding business can run normally after data migration.

[0085] In an embodiment of the present application, there may be both priorities and dependencies between multiple data to be migrated. This embodiment can configure the data migration order in the migration scheduling strategy based on the priorities and dependencies between the data. Exemplarily, the current data to be migrated includes data 1 to be migrated, data 2 to be migrated, and data 3 to be migrated. The priority of data 2 to be migrated is higher than that of data 1 to be migrated, and the priority of data 1 to be migrated is higher than that of data 3 to be migrated. There is a dependency relationship between data 2 to be migrated and data 3 to be migrated. If the data migration order in the migration scheduling strategy is configured only according to the priority, then when the data migration task is started, the data migration order is data 2 to be migrated, data 1 to be migrated, and data 3 to be migrated. During this migration method, if the migration network is unstable when migrating data 3, resulting in the failure of data 3 to be migrated, the corresponding business cannot operate normally at the data migration end. Based on this, when configuring the data migration order in the migration scheduling strategy, this embodiment sets the migration order by considering the priority and dependency of the data to be migrated at the same time, thereby effectively improving the reliability and stability of the corresponding business operation at the data migration end.

[0086] In the embodiments of the present application, Figure 5 As shown, the process of configuring the migration scheduling strategy according to the data to be migrated includes but is not limited to step S510 and step S520:

[0087] Step S510, calculating the amount of data to be migrated;

[0088] Step S520: configure the data migration mode in the migration scheduling strategy according to the data volume.

[0089] It is understandable that the different amounts of data to be migrated will affect the efficiency of data migration. Therefore, this embodiment calculates the amount of data to be migrated first, and then configures the data migration method in the migration scheduling strategy according to the amount of data, thereby effectively improving the migration efficiency of the data to be migrated. For example, when the current data to be migrated includes data 4 to be migrated, data 5 to be migrated, and data 6 to be migrated, when the amount of data 4 to be migrated is large, the data migration method corresponding to data 4 to be migrated can be configured as reduced migration; when the amounts of data 5 to be migrated and data 6 to be migrated are both small and mixed together without affecting each other, the data migration methods corresponding to data 5 to be migrated and data 6 to be migrated can be configured as mixed migration.

[0090] In the embodiment of the present application, the data migration mode in the migration scheduling strategy can also be configured by considering the business requirements corresponding to the data to be migrated. For example, when the data volume of the data 5 to be migrated is large and the business requirement is to run the business stably and accurately, the migration mode corresponding to the data 5 to be migrated can be configured as full migration, so as to ensure the accuracy of data migration, and thus enable the corresponding business to run accurately and stably; when the data volume of the data 6 to be migrated is small and the business requirement is to execute multiple identical sub-businesses simultaneously, the migration mode corresponding to the data 5 to be migrated can be configured as incremental migration, so that the data migration end contains multiple copies of the data 6 to be migrated, so that the businesses corresponding to multiple data 6 to be migrated can be executed synchronously.

[0091] In the embodiments of the present application, Figure 6 As shown, the process of configuring the migration scheduling strategy according to the data to be migrated includes but is not limited to step S610 and step S640:

[0092] Step S610: Acquire lifecycle information of the data to be migrated and business requirement information corresponding to the data to be migrated;

[0093] Step S620: analyzing the usage frequency of the data to be migrated;

[0094] Step S630: determining a correspondence relationship of the data to be migrated according to the life cycle information and the usage frequency, wherein the correspondence relationship is used to characterize a mapping relationship between a data partition and a data type of the data to be migrated;

[0095] Step S640: configure resource allocation information in the migration scheduling strategy according to the corresponding relationship and business demand information.

[0096] It is understandable that the life cycle information of the data to be migrated refers to the entire time period from the generation, use to the final abandonment or archiving of the data to be migrated. This life cycle can be obtained through statistics and analysis. Based on the life cycle information, the data to be migrated can be divided into different categories according to the length of the life cycle, such as long-term storage data, medium-term active data, and short-term temporary data. The frequency of use of the data to be migrated can be evaluated and calculated by monitoring indicators such as the number of accesses to the data, query time, and modification frequency. The frequency of use can reflect the activity and importance of the data to be migrated. High-frequency use of data usually means higher business value and more frequent access needs.

[0097] After obtaining the life cycle information and usage frequency of the data to be migrated, this embodiment determines the corresponding relationship of the data to be migrated according to the life cycle information and the usage frequency. Among them, the corresponding relationship is used to characterize the mapping relationship between the data partition and the data type of the data to be migrated. The data partition may refer to the storage area of ​​the data. Exemplarily, when the data type of the data to be migrated is determined to be a long-term storage data type according to the life cycle information, it can be determined that the data of this type has a low usage frequency, but because these data have important historical value or regulatory requirements need to be retained for a long time, therefore, the resource allocation information of this type of data can be configured as a low-cost storage medium, such as a cold storage or archiving system, to reduce the occupation of high-performance storage resources. When the data type of the data to be migrated is determined to be a medium-term active data type according to the life cycle information, it can be determined that this type of data to be migrated has a high usage frequency and is frequently accessed in the daily operation of the business system. Therefore, according to the configuration of the resource allocation information of this type of data, it is a high-performance storage medium, such as an SSD or a high-speed disk array, to ensure fast access to data and efficient operation of the business system. When it is determined based on the life cycle information that the data type of the data to be migrated is a short-term temporary data type, it can be determined that this type of data has an extremely short life cycle and a high degree of temporariness, such as temporary files, cached data, etc. The frequency of use of this type of data is low. Therefore, the resource allocation information of this type of data to be migrated can be configured as a temporary storage area or directly processed in the memory to reduce the occupation of memory resources.

[0098] It is understandable that after obtaining the correspondence of the data to be migrated, this embodiment can also configure the resource allocation information in the migration scheduling strategy in combination with the business demand information. Specifically, the resource allocation information includes but is not limited to computing resource information, storage resource information, and network resource information. Exemplarily, for the correspondence of high-performance storage media for medium-term active data types and when the business demand information is high operating efficiency, the resource allocation information in the migration scheduling strategy can be configured as idle network resource information and high storage resource information, so that data migration can be performed through the network corresponding to the idle network resource information, and the data to be migrated can be stored through the storage medium corresponding to the high storage resource information, thereby effectively improving the data migration efficiency and meeting the corresponding business needs.

[0099] In the embodiment of the present application, after obtaining the data to be migrated, the data to be migrated is preprocessed. Specifically, Figure 7 As shown, the preprocessing process of the embodiment of the present application includes but is not limited to step S710 to step S730:

[0100] Step S710: clean the data to be migrated to obtain first migration data;

[0101] Step S720: convert the format of the first migration data to obtain second migration data;

[0102] Step S730: verify the data verification identifier in the second migration data to obtain a preprocessing result.

[0103] It is understood that data cleaning includes but is not limited to abnormal data search, filtering, etc.; format conversion includes but is not limited to transmission format conversion, reception format conversion, etc. This embodiment cleans and converts the data to be migrated to ensure the feasibility, compatibility and compliance with the preset format requirements of the data to be migrated. In addition, after completing the data conversion, this embodiment verifies the data verification identifier in the second migration data after the data conversion, thereby ensuring the accuracy and security of the migrated data.

[0104] It is understandable that after completing the preprocessing of the data to be migrated and the configuration of the migration scheduling strategy, the real-time migration task can be started to realize the migration process of the data to be migrated. Among them, the execution process of the real-time migration task of this embodiment is executed according to the configuration of the migration scheduling strategy and the definition of the real-time migration task. During the data migration process, each type of data migration task will output corresponding data migration report information, which can be displayed in the human-computer interaction interface, so that the target object can view the migration status of the data to be migrated in real time, such as whether the data migration fails or succeeds. For the real-time migration task of successful data migration, the detailed statistical information corresponding to the real-time migration task can be obtained, such as the amount of data migrated, the time consumed, the average migration speed, etc., so that the migration efficiency and resource usage status information can be generated according to the detailed statistical information, and then the migration scheduling strategy can be optimized according to the migration efficiency and resource usage status information. Among them, the migration efficiency can be evaluated and generated according to the data migration duration, the amount of successfully migrated data, the accuracy of the data after migration, and other information. The resource usage status information may include but is not limited to the occupancy rate of network resources during the transmission process, the amount of resource occupancy after migration, etc. For example, if the network resource occupancy rate is only 50% during the data migration process, the allocation of network resources in the corresponding migration scheduling strategy can be reduced according to the network occupancy rate, so that the remaining network resources can be allocated to other required data migration processes to improve network resource utilization and data migration efficiency.

[0105] In the embodiments of the present application, Figure 8 As shown, taking XXL-JOB as a distributed task scheduling end as an example, the method provided by the embodiment of the present application includes but is not limited to steps S810 to S890 during the application process:

[0106] Step S810: The target object sets multiple scheduling strategies to be selected in the interface provided by XXL-JOB;

[0107] Step 820: The target object creates a real-time migration task in the interface provided by XXL-JOB, and enters the task name, data source location, and target data location corresponding to the real-time migration task. XXL-JOB determines the data to be migrated corresponding to the real-time migration task according to the task name and data source location.

[0108] Step S830, XXL-JOB verifies the data to be migrated, saves the verification result to the database, and updates the real-time migration task corresponding to the data to be migrated to a verified state;

[0109] Step S840, analyzing whether the verification result is successful. When the verification result is a failure, executing step S850; when the verification result is a success, executing step S860;

[0110] Step S850: output a failure result list, and update the real-time migration task corresponding to the data to be migrated to a verification failure state;

[0111] Step S860: Update the real-time migration task corresponding to the data to be migrated to a waiting-to-start state, waiting for the target object to be verified again;

[0112] Step S870: after the target object is successfully verified again, a migration scheduling policy for the current data to be migrated is configured from a plurality of scheduling policies to be selected according to the data to be migrated, and data migration logic is executed one by one according to the configured migration scheduling policy;

[0113] Step S880: Save the result of data migration failure to the database, and update the status of the real-time migration task to migration failure;

[0114] Step S890: Save the result of successful data migration to the database, and mark the execution as successful according to the status of the real-time migration task.

[0115] To sum up, the method provided in the embodiment of the present application can improve the flexibility and scalability of the business system in data migration by applying it to the distributed task scheduling end, and after configuring the migration scheduling strategy in the distributed task scheduling end according to the real-time migration task and the corresponding migration data, the migration data is pre-processed, thereby effectively improving the efficiency of the data migration process, the accuracy and completeness of the data migration.

[0116] Reference Fig. 9 The embodiment of the present application provides a data migration device based on task scheduling, the device is arranged at a distributed task scheduling end, the distributed task scheduling end interacts with multiple data ends, and the data ends include a data migration end and a data migration end. The device includes:

[0117] The first module 910 is used to obtain the real-time migration task of the data migration-out end and the data to be migrated corresponding to the real-time migration task;

[0118] The second module 920 is used to configure a migration scheduling strategy according to the data to be migrated;

[0119] The third module 930 is used to pre-process the data to be migrated and obtain a pre-processing result;

[0120] The fourth module 940 is used to execute the real-time migration task according to the preprocessing result and the migration scheduling strategy, so as to transfer the data to be migrated from the data migration-out end to the data migration-in end.

[0121] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data migration method based on task scheduling when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0123] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0125] The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0126] The memory 1020 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 calls and executes the data migration method based on task scheduling in the embodiment of this application;

[0127] Input / output interface 1030, used to implement information input and output;

[0128] The communication interface 1040 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0129] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0130] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .

[0131] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data migration method based on task scheduling is implemented.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0135] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0136] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0138] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0144] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data migration method based on task scheduling, characterized in that: The method is applied to a distributed task scheduling end, the distributed task scheduling end interacts with multiple data ends, the data ends include a data migration end and a data migration end, and the method includes the following steps: Acquire the real-time migration task of the data migration-out end and the to-be-migrated data corresponding to the real-time migration task; Configure a migration scheduling strategy according to the data to be migrated; Preprocessing the data to be migrated to obtain a preprocessing result; The real-time migration task is executed according to the preprocessing result and the migration scheduling strategy to transfer the data to be migrated from the data migration-out end to the data migration-in end.

2. The method according to claim 1, characterized in that The configuring a migration scheduling strategy according to the data to be migrated includes: Obtaining business scenario information corresponding to the data to be migrated; The data migration timing information in the migration scheduling strategy is configured according to the business scenario information.

3. The method according to claim 1, characterized in that: The configuring a migration scheduling strategy according to the data to be migrated includes: Acquire the priority of the data to be migrated and the dependency relationship between the plurality of data to be migrated; The data migration order in the migration scheduling policy is configured according to the priority and the dependency relationship.

4. The method according to claim 1, characterized in that: The configuring a migration scheduling strategy according to the data to be migrated includes: Calculating the amount of data to be migrated; The data migration mode in the migration scheduling strategy is configured according to the data volume.

5. The method according to claim 1, characterized in that The configuring a migration scheduling strategy according to the data to be migrated includes: Acquire lifecycle information of the data to be migrated and business requirement information corresponding to the data to be migrated; Analyzing the usage frequency of the data to be migrated; determining a correspondence relationship of the data to be migrated according to the life cycle information and the usage frequency, wherein the correspondence relationship is used to characterize a mapping relationship between a data partition and a data type of the data to be migrated; The resource allocation information in the migration scheduling strategy is configured according to the corresponding relationship and the business requirement information.

6. The method according to claim 1, characterized in that The preprocessing of the data to be migrated to obtain a preprocessing result includes: Cleaning the data to be migrated to obtain first migration data; performing format conversion on the first migration data to obtain second migration data; The data verification identifier in the second migration data is verified to obtain the preprocessing result.

7. The method according to claim 1, characterized in that When executing the real-time migration task according to the preprocessing result and the migration scheduling strategy, the method further includes the following steps: Obtaining data migration report information corresponding to the real-time migration task; Generate migration efficiency and resource usage status information according to the data migration report information; The migration scheduling strategy is optimized according to the migration efficiency and the resource usage status information.

8. A data migration device based on task scheduling, characterized in that: The device is arranged at a distributed task scheduling end, the distributed task scheduling end interacts with a plurality of data ends, the data ends include a data migration end and a data migration end, and the device includes: The first module is used to obtain the real-time migration task of the data migration-out end and the data to be migrated corresponding to the real-time migration task; The second module is used to configure a migration scheduling strategy according to the data to be migrated; The third module is used to preprocess the data to be migrated to obtain a preprocessing result; The fourth module is used to execute the real-time migration task according to the preprocessing result and the migration scheduling strategy to transfer the data to be migrated from the data migration-out end to the data migration-in end.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images