Data migration method and device, storage medium and electronic device

Abstract

The application discloses a data migration method and device, a storage medium and an electronic device. It relates to the field of big data. The method comprises the following steps: obtaining M to-be-migrated tables contained in a target transmission task in a task list, and obtaining a data filtering strategy of a database; removing part of the data in each to-be-migrated table in sequence according to the data filtering strategy, obtaining M candidate tables, and generating P first files; determining the task type of the target transmission task; in the case where the task type is a timing task, sequentially sending each first file to a target storage table in a target data lake; in the case where the task type is a temporary task, merging the first files with the same file name to obtain N second files, and sequentially sending each second file to the target storage table of the first file in the target data lake. Through the application, the problem of poor transmission timeliness of data transmission using a batch file transmission method in the related art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, the field of financial technology and other related fields, in particular, to a data migration method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the gradual increase of business data in financial institutions, there will be a scenario of storing data tables from business systems to data lakes, for example, migrating data of an Oracle database to a Hadoop data warehouse, so as to achieve the effect of long-term storage of data.

[0003] Currently, when data migration is performed, a batch file transmission method is usually used. However, the batch lake method cannot timely import real-time generated data into the lake, which causes downstream data users to be unable to obtain the latest data, thereby affecting the timely use of data.

[0004] In view of the poor transmission timeliness of the data transmission using the batch file transmission method in the related art, an effective solution has not been proposed yet. SUMMARY

[0005] The present application provides a data migration method and device, a storage medium and an electronic device to solve the problem of poor transmission timeliness of data transmission using the batch file transmission method in the related art.

[0006] According to one aspect of the present application, a data migration method is provided. The method comprises: obtaining M target transmission tasks in a task list, wherein M is a positive integer, and each target transmission task contains a table to be migrated; obtaining a data filtering strategy of a database from the database to which the M tables to be migrated belong, wherein each table to be migrated contains data; removing part of the data in each table to be migrated according to the data filtering strategy to obtain M candidate tables, and generating a first file according to the data in each candidate table to obtain P first files; determining the type of the target transmission task, wherein the type of the task includes a temporary task and a timing task; in the case of the type of the task being a timing task, sending each first file to a target storage table of the first file in a target data lake; in the case of the type of the task being a temporary task, obtaining the file name of each first file, merging the first files with the same file name to obtain N second files, and sending each second file to the target storage table of the first file in the target data lake, wherein N is a positive integer.

[0007] Optionally, before the step of generating the first file according to the data in each candidate table to obtain P first files, the method further comprises: obtaining a preset file generation time of the target transmission task, and obtaining a current time, wherein the current time is a time when the step of obtaining the M tables to be migrated in the target transmission task in the task list is performed; determining whether the preset file generation time and the current time are the same; in the case that the preset file generation time and the current time are the same, performing the step of generating the first file according to the data in each candidate table to obtain P first files; in the case that the preset file generation time and the current time are not the same, suspending the execution of the target transmission task until the current time and the preset file generation time are the same.

[0008] Optionally, after the step of generating the first file according to the data in each candidate table to obtain P first files, the method further comprises: sequentially verifying each first file, and determining whether there is an abnormal file in the P first files according to the verification result; in the case that there is no abnormal file in the P first files, recording the generation result of each file in the file execution detail table, and performing the step of determining the task type of the transmission task of the M tables to be migrated; in the case that there is an abnormal file in the P first files, recording the generation result of each file in the file execution detail table, sending the abnormal file to the server, and performing the step of determining the task type of the target transmission task according to the remaining non-abnormal first file.

[0009] Optionally, the method further comprises: detecting whether there is a target table that has undergone a table structure change in the plurality of initial tables stored in the database; in the case that there is a target table, obtaining table structure change content of the target table, sending the table structure change content to the target data lake, and recording a first preset time in the target table, wherein the table structure of the target storage table corresponding to the target table is changed according to the table structure change content, and the first preset time represents the latest change time of the target storage table corresponding to the target table.

[0010] Optionally, after the step of obtaining the table structure change content of the target table, the method further comprises: obtaining filtering content related to the target table from the data filtering strategy, and determining whether the table structure change of the target table affects the filtering content, wherein the filtering content indicates the data content used to generate the first file; in the case that the table structure change does not affect the filtering content, canceling the step of sending the table structure change content to the target data lake and recording the first preset time in the target table.

[0011] Optionally, in the case that the task type is a timing task, the first file is generated according to the data in each candidate table in sequence, and P first files are obtained, including: for each candidate table, determining whether the table structure of the first to-be-migrated table corresponding to the candidate table is changed; in the case that the table structure is changed, determining whether the current time is earlier than the second preset time recorded in the candidate table, wherein the current time is the time at which the step of obtaining the M to-be-migrated tables included in the target transmission task in the task list is performed, and the second preset time represents the latest change time of the target storage table corresponding to the candidate table; in the case that the current time is not earlier than the second preset time, generating the first file according to the data in the candidate table; in the case that the current time is earlier than the second preset time, obtaining the change content of the first to-be-migrated table, and restoring the first to-be-migrated table according to the change content to obtain a second to-be-migrated table; regenerating the candidate table through the second to-be-migrated table, and generating the first file according to the data in the regenerated candidate table.

[0012] Optionally, in the case that the table structure is changed, the method further comprises: obtaining the filtering content related to the candidate table from the data filtering strategy, and determining whether the change of the table structure of the candidate table affects the filtering content, wherein the filtering content indicates the data content used to generate the first file; in the case that the change of the table structure does not affect the filtering content, performing the step of generating the first file according to the data in the candidate table.

[0013] According to another aspect of the present application, a data migration device is provided. The device comprises: a first obtaining unit configured to obtain M to-be-migrated tables included in a target transmission task in a task list, and obtain a data filtering strategy of a database from the database to which the M to-be-migrated tables belong, wherein M is a positive integer, and there is data in each to-be-migrated table; a removing unit configured to remove part of the data in each to-be-migrated table in sequence through the data filtering strategy to obtain M candidate tables, and generate a first file according to the data in each candidate table in sequence to obtain P first files; a determining unit configured to determine the task type of the target transmission task, wherein the task type includes a temporary task and a timing task; a first sending unit configured to, in the case that the task type is a timing task, sequentially send each first file to a target storage table of the first file in a target data lake; and a second sending unit configured to, in the case that the task type is a temporary task, obtain the file name of each first file, and merge the first files with the same file name to obtain N second files, and sequentially send each second file to the target storage table of the first file in the target data lake, wherein N is a positive integer.

[0014] According to another aspect of the present application, there is also provided a computer storage medium for storing a program, wherein the program, when executed, controls a device in which the computer storage medium is located to perform a data migration method.

[0015] According to another aspect of the present application, there is also provided an electronic device comprising one or more processors and a memory, wherein the memory stores computer readable instructions, and the processor is configured to execute the computer readable instructions, and wherein the computer readable instructions, when executed, perform a data migration method.

[0016] According to the present application, the following steps are adopted: obtaining M to-be-migrated tables contained in a target transmission task in a task list, and obtaining a data filtering strategy of a database to which the M to-be-migrated tables belong, wherein each to-be-migrated table contains data; removing part of the data in each to-be-migrated table in sequence according to the data filtering strategy to obtain M candidate tables, and generating a first file according to the data in each candidate table in sequence to obtain P first files; determining a task type of the target transmission task, wherein the task type includes a temporary task and a timing task; in the case that the task type is the timing task, sequentially sending each first file to a target storage table of the first file in a target data lake; in the case that the task type is the temporary task, obtaining a file name of each first file, and merging first files with the same file name to obtain N second files, and sequentially sending each second file to the target storage table of the first file in the target data lake. The problem of poor transmission timeliness of data transmission using a batch file transmission manner in the related art is solved. By filtering the data in the obtained transmission task, the data that needs to be transmitted is obtained, and it is determined according to the task type of the transmission task whether the transmission task is a timing task or a temporary task. In the case of the timing task, since the file is a large file in the table, the transmission can be directly performed. In the case of the temporary task, a plurality of small files are merged, thereby reducing the number of files. The timing task and the temporary task are processed separately, thereby achieving the effect of improving the data transmission timeliness. Moreover, the plurality of small files are merged into a large file, thereby achieving the effect of reducing the file receiving pressure of the receiving end. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and are not intended to be an improper limitation of the present application. In the drawings:

[0018] Figure 1 is a flowchart of a data migration method provided according to an embodiment of the present application;

[0019] Figure 2is a flowchart of an optional data migration method according to an embodiment of the present application;

[0020] Figure 3 is a flowchart of an execution process of a temporary task according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a data migration device according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties. For example, an interface is provided between the system and the relevant user or institution. Before obtaining the relevant information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information fed back by the aforementioned user or institution, the relevant information is obtained.

[0027] It should be noted that the data migration method, device, storage medium and electronic device determined by the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the data migration method, device, storage medium and electronic device determined by the present disclosure is not limited.

[0028] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] For the convenience of description, part of the nouns or terms related to the embodiments of the present application are described as follows:

[0030] Oracle database: a relational database management system.

[0031] Hadoop system: a system constructed based on a distributed system infrastructure.

[0032] Data lake: a centralized storage library that can accommodate relational data in any format from various data sources.

[0033] According to the embodiments of the present application, a data migration method is provided.

[0034] Figure 1 is a flowchart of the data migration method provided by the embodiments of the present application. As shown in Figure 1 , the method comprises the following steps:

[0035] Step S101, obtaining M to-be-migrated tables contained in a target transmission task in a task list, and obtaining a data filtering strategy of a database from the database to which the M to-be-migrated tables belong, wherein M is a positive integer, and there is data in each to-be-migrated table.

[0036] Specifically, the execution subject of the present application can be a Hadoop platform. The data transmission scheduling program in the Hadoop platform obtains a target transmission task in a task list. There can be multiple transmission tasks in the task list. After adding a transmission task in the task list, the data transmission scheduling program first obtains M to-be-migrated tables in the task, and obtains a data filtering strategy in a business system connected with the Hadoop platform, so that the data in the table can be processed according to the data filtering strategy.

[0037] It should be noted that each Hadoop platform can be connected with one business system, or can be connected with multiple business systems, but the multiple tables to be migrated in one transmission task belong to the same business system, and the multiple tables to be migrated in each transmission task are sent to the Hadoop platform from the Oracle database in the business system, or are added to the transmission task by manual input.

[0038] It should be noted that the data filtering strategy is used to delete the data in the table to be migrated that does not need to be migrated, thereby reducing the amount of data to be transmitted. At the same time, the data filtering strategy can be included in the preset parameter table, and the preset parameter table can be used to control the entire transmission process. For example, the preset parameter table can include data filtering strategy, target storage table of storage file, data type, transmission batch, and the like, so that the Hadoop platform can perform data migration operation through the information in the preset parameter table.

[0039] In step S102, part of the data in each table to be migrated is removed in sequence through the data filtering strategy, M candidate tables are obtained, and P first files are obtained by generating the first file according to the data in each candidate table.

[0040] Specifically, after obtaining the data filtering strategy, the data to be deleted in each table to be migrated can be removed according to the data filtering strategy to obtain an updated table, that is, a candidate table, and a first file can be generated according to the data in the candidate table, so that data migration can be performed in the form of encapsulation and data format of the first file.

[0041] For example, the table to be migrated A includes 10 columns of data, and the data filtering strategy shows that the columns a, b, c, d, and e in the table to be migrated A need to be migrated, at this time, the remaining 5 columns can be deleted to obtain the candidate table A.

[0042] It should be noted that when generating a file from a table, the data in one table can generate one file, and can also generate multiple files, or can not generate a corresponding file, therefore, there is no fixed number corresponding relationship between P and M.

[0043] In step S103, the task type of the target transmission task is determined, wherein the task type includes a temporary task and a timing task.

[0044] Specifically, after the file generation is completed, the task type of the target transmission task can be determined, and the transmission mode can be determined according to the task type.

[0045] It should be noted that the step of determining the task type of the target transmission task can be performed after generating the file, or after obtaining the M to-be-migrated tables contained in the target transmission task in the task list. In the case that the task type is a timing task and the task execution time has not been reached, the first file can not be produced first, and the production can be performed after the execution time is reached.

[0046] In step S104, in the case that the task type is a timing task, each first file is sent to the target storage table of the first file in the target data lake in sequence.

[0047] Specifically, in the case that the task type is a timing task, if the current time is the same as the timing time of the task, the transmission of the plurality of first files can be started. The target storage table to which the plurality of first files are transmitted is the target storage table corresponding to the to-be-migrated table to which each first file belongs in the data lake, so as to complete the data transmission.

[0048] In the case that the timing time has not been reached, it is indicated that the current time cannot execute the data migration task, and therefore waiting is needed. In the case that the timing time is reached, the data transmission can be performed. The timing time can be a system idle time, for example, 02:00, 12:00, 18:00, etc., so as to ensure that the influence of the data migration operation on the system is small.

[0049] In step S105, in the case that the task type is a temporary task, the file name of each first file is obtained, and the first files with the same file name are merged to obtain N second files, and each second file is sent to the target storage table of the first file in the target data lake in sequence, wherein N is a positive integer.

[0050] Specifically, in the case that the task type is a temporary task, it is indicated that the data at this time is relatively urgent data manually input by a worker. At this time, the data needs to be transmitted immediately. However, since the temporary task can be composed of a plurality of small data, different small data can belong to the same table. Therefore, after generating a plurality of first files, it can be determined according to the file name which files belong to the same table, and the files are merged. The plurality of second files after merging are transmitted, so as to reduce the number of files in the temporary task and ensure the timeliness of the file transmission.

[0051] The data migration method provided in the embodiments of the present application comprises the following steps: obtaining M to-be-migrated tables contained in a target transmission task in a task list, and obtaining a data filtering strategy of a database to which the M to-be-migrated tables belong, wherein, there is data in each to-be-migrated table; removing part of the data in each to-be-migrated table in sequence according to the data filtering strategy to obtain M candidate tables, and generating a first file according to the data in each candidate table to obtain P first files; determining a task type of the target transmission task, wherein the task type comprises a temporary task and a timing task; in the case that the task type is the timing task, sequentially sending each first file to a target storage table of the first file in a target data lake; in the case that the task type is the temporary task, obtaining a file name of each first file, and merging the first files with the same file name to obtain N second files, and sequentially sending each second file to the target storage table of the first file in the target data lake. The problem of poor transmission timeliness of data transmission using a batch file transmission manner in the related art is solved. By filtering the data in the obtained transmission task, the data that needs to be transmitted is obtained, and it is determined according to the task type of the transmission task whether the transmission task is a timing task or a temporary task. In the case of the timing task, since the file is a large file in the table, the transmission can be directly performed. In the case of the temporary task, a plurality of small files are merged, thereby reducing the number of files. The timing task and the temporary task are processed separately, thereby achieving the effect of improving the data transmission timeliness, and the plurality of small files are merged into a large file, thereby achieving the effect of reducing the file receiving pressure of the receiving end.

[0052] Optionally, in the data migration method provided in the embodiments of the present application, before the step of generating a first file according to the data in each candidate table to obtain P first files, the method further comprises the following steps: obtaining a preset file generation time of the target transmission task, and obtaining a current time, wherein the current time is the time when the step of obtaining M to-be-migrated tables contained in a target transmission task in a task list is performed; determining whether the preset file generation time is the same as the current time; in the case that the preset file generation time is the same as the current time, performing the step of generating a first file according to the data in each candidate table to obtain P first files; in the case that the preset file generation time is not the same as the current time, suspending the execution of the target transmission task until the current time is the same as the preset file generation time.

[0053] Specifically, in the case that the task type is a timing task, a preset file generation time is set in the target transmission task, that is, the table generation file in the target transmission task, in the case that the time of receiving the target transmission task does not reach the preset file generation time, for example, the time of obtaining the M to-be-migrated tables contained in the target transmission task in the task list is 15:00, and the preset file generation time of the task is 18:00, at this time, the preset file generation time is not reached, therefore, the target transmission task can be shelved until 18:00, and at 18:00, the file generation and sending operation of the table in the target transmission task is performed, so as to ensure that the timing task can be executed at the preset idle time.

[0054] It should be noted that since the file will be sent immediately after the table generation file is generated, the time of generating the file and the time of sending the file can be regarded as the same time. If the file is generated by mistake in the case that the preset file generation time is not reached at the current time, at the time of sending the file, it is also necessary to determine again whether the current time is the preset file generation time, and in the case that it is not, the sending operation of the file is suspended, so as to ensure that the file is sent at the correct time.

[0055] In order to ensure the accuracy of the file, optionally, in the data migration method provided in the embodiments of the present application, after generating the first file according to the data in each candidate table in sequence to obtain P first files, the method further comprises: checking each first file in sequence, and confirming whether there is an abnormal file in the P first files according to the checking result; in the case that there is no abnormal file in the P first files, recording the generation result of each file in the file execution detail table, and performing the step of determining the task type of the transmission task of the M to-be-migrated tables; in the case that there is an abnormal file in the P first files, recording the generation result of each file in the file execution detail table, sending the abnormal file to the server, and performing the step of determining the task type of the target transmission task according to the remaining non-abnormal first files.

[0056] Specifically, after generating the first file, since there are multiple to-be-migrated tables in the task, each table corresponds to a first file, therefore, in order to ensure the accuracy of the first file, it is necessary to check each first file, so as to ensure the correctness of the first file.

[0057] After checking, if there is an abnormal file, the generation result of each file is recorded in the file execution detail table, at the same time, the abnormal file is sent to the server, and the remaining normal files are normally transmitted, the operation and maintenance personnel can confirm the abnormal reason of the abnormal file through the server, and judge whether the abnormal reason is that the to-be-migrated table is abnormal or that the file is generated abnormally.

[0058] It should be noted that, regardless of whether there is an abnormal file, the generation result of each file needs to be added to the file execution details table, so as to record the generation results of all files, facilitating subsequent review.

[0059] Optionally, in the data migration method provided in the embodiments of the present application, the method further comprises: detecting whether a target table structure that has changed is present in a plurality of initial tables stored in the database; in the case where the target table structure is present, obtaining table structure change content of the target table structure, sending the table structure change content to the target data lake, and recording a first preset time in the target table structure, wherein a table structure of a target storage table corresponding to the target table structure is changed according to the table structure change content, and the first preset time represents the latest change time of the target storage table corresponding to the target table structure.

[0060] Specifically, since the table structure of the to-be-migrated table in the business system may change with the increase of business needs and data volume, and since the table structure of the target storage table in the data lake needs to be consistent with the table structure of the corresponding to-be-migrated table, so that the data can be correctly stored, when the table structure of the target table changes, the change content needs to be sent to the data lake, so that the corresponding target storage table in the data lake is changed according to the change content, so that the transmitted data can be correctly stored in the target storage table.

[0061] It should be noted that the database of the business system stores a plurality of initial tables, and when the data in the initial tables changes and needs to be migrated, the initial table is referred to as a to-be-migrated table.

[0062] It should be noted that there is a time difference between the business system and the data lake, that is, the table structure in the business system changes, while the table structure in the data lake does not change. At this time, if data migration is performed, it will cause the data to be unable to be correctly written into the target storage table. Therefore, the first preset time needs to be set according to the time difference, and the first preset time is the time when the target storage table completes the table structure change at the latest. When the first preset time is reached, the target storage table must be able to complete the change of the table structure. Therefore, the first preset time needs to be set according to the time difference between the business system and the data lake, and the first preset time is stored in the target table. Thus, when the file is generated according to the target table, it can be judged whether the first file can be generated according to the current table structure of the target table.

[0063] Optionally, in the data migration method provided in the embodiments of the present application, after obtaining the table structure change content of the target table, the method further comprises: obtaining filtering content related to the target table from the data filtering strategy, and judging whether the table structure change of the target table affects the filtering content, wherein the filtering content indicates the data content used to generate the first file; in the case where the table structure change does not affect the filtering content, canceling the step of sending the table structure change content to the target data lake and recording the first preset moment in the target table.

[0064] Specifically, since the change frequency of the table structure of the target table in the business system is high, the change frequency of the target storage table in the data lake is also high. Therefore, in order to reduce the change frequency of the structure of the target storage table in the data lake, after the table structure of the to-be-migrated table is changed, the filtering content related to the target table is obtained from the data filtering strategy to judge whether the table structure change affects the filtering content. In the case where the change content does not affect the filtering content, the table structure of the target storage table in the data lake can not be changed, thereby reducing the change frequency of the table structure of the target storage table.

[0065] For example, the target table B includes 10 rows and 10 columns of data, and after the table structure change, the target table B is changed to 10 rows and 20 columns of data. At this time, the data filtering strategy of the target table B is to retain the data of rows 1-5 and delete the remaining data. In the case of removing the data of the target table B before the change and the target table B after the change, the candidate table obtained after the data removal is still the same. At this time, the table structure of the target storage table corresponding to the target table B in the data lake does not need to be modified.

[0066] Optionally, in the data migration method provided in the embodiments of the present application, in the case where the task type is a timing task, the first file is generated according to the data in each candidate table in turn to obtain P first files, comprising: for each candidate table, judging whether the table structure of the first to-be-migrated table corresponding to the candidate table is changed; in the case where the table structure is changed, judging whether the current moment is earlier than the second preset moment recorded in the candidate table, wherein the current moment is the moment of executing the step of obtaining the M to-be-migrated tables contained in the target transmission task in the task list, and the second preset moment represents the latest change moment of the target storage table corresponding to the candidate table; in the case where the current moment is not earlier than the second preset moment, generating the first file according to the data in the candidate table; in the case where the current moment is earlier than the second preset moment, obtaining the change content of the first to-be-migrated table, and restoring the first to-be-migrated table according to the change content to obtain a second to-be-migrated table; regenerating the candidate table through the second to-be-migrated table, and generating the first file according to the data in the regenerated candidate table.

[0067] Specifically, when generating the file, it is first needed to determine whether the first to-be-migrated table corresponding to the file has undergone table structure change in the time range between the last migration data and the current migration data, in the case where no table structure change has occurred, the file can be directly generated, in the case where table structure change has occurred, the second preset moment in the to-be-migrated table is obtained, and it is determined whether the moment when the file is currently generated is earlier than the second preset moment, in the case where the moment when the file is currently generated is earlier than the second preset moment, it indicates that the table structure of the target storage table corresponding to the first to-be-migrated table has not been changed, at this time, the table structure of the first to-be-migrated table is restored to before the change to obtain a second to-be-migrated table, and the first file is generated according to the second to-be-migrated table, so as to ensure that the target storage table can correctly receive the first file and enter the data in the file into the table.

[0068] Optionally, in the data migration method provided in the embodiments of the present application, in the case where the table structure is changed, the method further comprises: obtaining the screening content related to the candidate table from the data screening strategy, and determining whether the change of the table structure of the candidate table affects the screening content, wherein the screening content indicates the data content used to generate the first file; in the case where the change of the table structure does not affect the screening content, performing the step of generating the first file according to the data in the candidate table.

[0069] Specifically, in the case where the table structure is changed, if the change of the table structure of the candidate table does not affect the screening content, it indicates that the file obtained after the data is removed is the same as the file before the table structure is changed, at this time, it is not needed to determine whether the current moment is earlier than the second preset moment, and the first file can be directly generated, at this time, the first file obtained can be normally received by the target storage table.

[0070] Figure 2 is a flowchart of the optional data migration method provided in the embodiments of the present application, as shown in Figure 2 the target transmission task in the task list is obtained, and in the case where the task is a timing task, the data in each to-be-migrated table in the target transmission task is removed through the data screening strategy in the parameter table to obtain a plurality of candidate tables.

[0071] For any one candidate table, it is judged whether the candidate table has table structure change, in the case of table structure change, it is judged whether the current time is earlier than the preset time, in the case of earlier than the preset time, the target storage table is not completed table structure change, at this time, the candidate table corresponding to the to-be-migrated table is recovered according to the change content, and the candidate table is regenerated from the recovered table, and the file is generated according to the regenerated candidate table, in the case of not earlier than the preset time, the file can be directly generated according to the current candidate table, after successfully generating the file, it is recorded in the task execution detail table, and the file is sent to the target storage table in the data lake, in the case of not successfully generating the file, it is recorded in the task execution detail table, and the exception file is sent to the server. Thus, the file migration operation under the timing task is completed.

[0072] Figure 3 The execution flow chart of the optional temporary task is provided according to the embodiments of the application, as shown in Figure 3 After obtaining the target transmission task, and in the case of the task being a timing task, a first file is generated according to a plurality of to-be-migrated tables, after successfully generating the first file, the same name small file is merged, the online interface is called to synchronize the merged file to the target storage table in the data lake, thereby completing the execution of the temporary task.

[0073] Thus, through the above two processes, the data in the obtained transmission task is screened to obtain the data that needs to be transmitted, and it is determined according to the task type of the transmission task whether the transmission task is a timing task or a temporary task, in the case of the timing task, the file is a large file in the table, so the transmission can be directly performed, in the case of the task being a temporary task, a plurality of small files are merged, thereby reducing the number of files, the timing task and the temporary task are processed separately, thereby achieving the effect of improving the timeliness of data transmission, and the plurality of small files are merged into a large file, thereby reducing the file receiving pressure of the receiving end.

[0074] It should be noted that the data transmitted by the timing task can be stored in the target storage table in the public library in the data lake, and the data transmitted by the temporary task can be stored in the target storage table in the private library in the data lake, the data lake can merge the data in the corresponding two target storage tables in the public library and the private library, and set access permissions, so that the data in the table can be accessed through the preset access method.

[0075] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in different order.

[0076] The embodiment of the present application further provides a data migration device. It should be noted that the data migration device of the embodiment of the present application can be used to execute the data migration method provided by the embodiment of the present application. The data migration device provided by the embodiment of the present application is introduced as follows.

[0077] Figure 4 FIG. 1 is a schematic diagram of a data migration device according to the embodiment of the present application. As shown in the figure, the device comprises a first acquisition unit 41, a removal unit 42, a determination unit 43, a first sending unit 44 and a second sending unit 45. Figure 4

[0078] The first acquisition unit 41 is configured to acquire M to-be-migrated tables contained in a target transmission task in a task list, and acquire a data filtering strategy of a database to which the M to-be-migrated tables belong from the database, wherein M is a positive integer, and each to-be-migrated table contains data.

[0079] The removal unit 42 is configured to remove part of the data in each to-be-migrated table according to the data filtering strategy to obtain M candidate tables, and generate a first file according to the data in each candidate table to obtain P first files.

[0080] The determination unit 43 is configured to determine a task type of the target transmission task, wherein the task type comprises a temporary task and a timing task.

[0081] The first sending unit 44 is configured to send each first file to a target storage table of the first file in a target data lake in a case where the task type is the timing task.

[0082] The second sending unit 45 is configured to acquire a file name of each first file, and merge the first files with the same file name to obtain N second files in a case where the task type is the temporary task, and send each second file to the target storage table of the first file in the target data lake in sequence, wherein N is a positive integer.

[0083] ​The data migration device provided in the embodiments of the present application obtains M to-be-migrated tables contained in a target transmission task in a task list through a first obtaining unit 41, and obtains a data filtering strategy of a database to which the M to-be-migrated tables belong, wherein M is a positive integer, and each to-be-migrated table contains data; a removing unit 42 removes part of the data in each to-be-migrated table according to the data filtering strategy to obtain M candidate tables, and generates a first file according to the data in each candidate table to obtain P first files; a determining unit 43 determines a task type of the target transmission task, wherein the task type includes a temporary task and a timing task; a first sending unit 44 sends each first file to a target storage table of the first file in a target data lake in a case where the task type is the timing task; and a second sending unit 45 obtains a file name of each first file and merges first files with the same file name to obtain N second files in a case where the task type is the temporary task, and sends each second file to the target storage table of the first file in the target data lake, wherein N is a positive integer. The problem of poor transmission timeliness of data transmission using a batch file transmission manner in the related art is solved. The data in the obtained transmission task is filtered to obtain data that needs to be transmitted, and it is determined according to the task type of the transmission task whether the transmission task is a timing task or a temporary task. In the case of the timing task, the file is a large file in the table, and therefore the transmission can be directly performed. In the case of the temporary task, a plurality of small files are merged, thereby reducing the number of files. The timing task and the temporary task are processed separately, thereby improving the data transmission timeliness, and the plurality of small files are merged into a large file, thereby reducing the file receiving pressure of the receiving end.

[0084] Optionally, in the data migration device provided in the embodiments of the present application, the device further includes: a second obtaining unit, configured to obtain a preset file generation moment of the target transmission task and obtain a current moment, wherein the current moment is a moment at which the step of obtaining M to-be-migrated tables contained in the target transmission task in the task list is performed; a first determining unit, configured to determine whether the preset file generation moment and the current moment are the same; a first executing unit, configured to perform the step of sending each first file to a target storage table of the first file in a target data lake in a case where the preset file generation moment and the current moment are the same; and a storage unit, configured to store the P first files to a cache area in a case where the preset file generation moment and the current moment are not the same until the current moment and the preset file generation moment are the same.

[0085] Optionally, in the data migration apparatus provided by the embodiment of the present application, the apparatus further comprises: a checking unit, configured to check each first file in sequence, and confirm whether there is an abnormal file in the P first files according to a checking result; a second execution unit, configured to, in the case that there is no abnormal file in the P first files, record a generation result of each file in a file execution detail table, and perform the step of determining the task type of the transmission task of the M tables to be migrated; and a third sending unit, configured to, in the case that there is an abnormal file in the P first files, record the generation result of each file in the file execution detail table, send the abnormal file to the server, and perform the step of determining the task type of the target transmission task according to the remaining non-abnormal first files.

[0086] Optionally, in the data migration apparatus provided by the embodiment of the present application, the apparatus further comprises: a detection unit, configured to detect whether there is a target table that has a table structure change in the plurality of initial tables stored in the database; and a third acquisition unit, configured to, in the case that there is the target table, acquire table structure change content of the target table, send the table structure change content to the target data lake, and record a first preset time in the target table, wherein the table structure of a target storage table corresponding to the target table is changed according to the table structure change content, and the first preset time represents the latest change time of the target storage table corresponding to the target table.

[0087] Optionally, in the data migration apparatus provided by the embodiment of the present application, the apparatus further comprises: a second judgment unit, configured to acquire filtering content related to the target table from the data filtering strategy, and judge whether the table structure change of the target table affects the filtering content, wherein the filtering content indicates data content used to generate the first file; and a canceling unit, configured to, in the case that the table structure change does not affect the filtering content, cancel the step of sending the table structure change content to the target data lake and recording the first preset time in the target table.

[0088] Optionally, in the data migration device provided by the embodiment of the present application, the removing unit 42 comprises: a first judging module, configured to judge, for each candidate table, whether the table structure of the first to-be-migrated table corresponding to the candidate table is changed; a second judging module, configured to judge, in the case that the table structure is changed, whether the current time is earlier than the second preset time recorded in the candidate table, wherein the current time is the time when the step of obtaining the M to-be-migrated tables contained in the target transmission task in the task list is performed, and the second preset time represents the latest change time of the target storage table corresponding to the candidate table; a first generating module, configured to generate the first file according to the data in the candidate table in the case that the current time is not earlier than the second preset time; an obtaining module, configured to obtain the change content of the first to-be-migrated table and restore the first to-be-migrated table according to the change content to obtain the second to-be-migrated table in the case that the current time is earlier than the second preset time; and a second generating module, configured to regenerate the candidate table through the second to-be-migrated table and generate the first file according to the data in the regenerated candidate table.

[0089] Optionally, in the data migration device provided by the embodiment of the present application, the device further comprises: a fourth obtaining unit, configured to obtain the screening content related to the candidate table from the data screening strategy and judge whether the change of the table structure of the candidate table affects the screening content, wherein the screening content indicates the data content used to generate the first file; and a third executing unit, configured to execute the step of generating the first file according to the data in the candidate table in the case that the change of the table structure does not affect the screening content.

[0090] The data migration device comprises a processor and a memory, the first obtaining unit 41, the removing unit 42, the determining unit 43, the first sending unit 44, the second sending unit 45 and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the program units stored in the memory.

[0091] The processor comprises a core, and the core calls the corresponding program units from the memory. The core can be set as one or more, and the problem of poor transmission timeliness in the related art using the batch file transmission mode for data transmission is solved by adjusting the core parameters.

[0092] The memory can comprise a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory comprises at least one memory chip.

[0093] The embodiment of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the data migration method.

[0094] The embodiment of the present application provides a processor, which is used for running a program, wherein the data migration method is executed when the program is running.

[0095] Figure 5 As shown in a schematic diagram of an electronic device provided by the embodiment of the present application, Figure 5 As shown in a schematic diagram of an electronic device provided by the embodiment of the present application, the electronic device 50 comprises a processor, a memory, and a program stored in the memory and capable of running on the processor, and the processor implements the steps of the data migration method when executing the program. The device herein can be a server, a PC, a PAD, a mobile phone, or the like.

[0096] The present application further provides a computer program product, which is suitable for executing a program initialized with the steps of the data migration method when executed on a data processing device.

[0097] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0098] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The function specified in one block or multiple blocks.

[0099] These computer program instructions can also be stored in a computer readable storage medium capable of guiding the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The function specified in one block or multiple blocks.

[0100] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0101] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0102] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory can also include a compact disk read only memory (CD-ROM), digital versatile disk (DVD), Blu-ray, or another non-transitory computer readable medium, which is non-volatile and non-transitory in nature, but volatile in that it can lose its content if the power to the computer is turned off or if the computer crashes. The memory is an example of a computer readable medium.

[0103] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0104] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0105] ​​The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A data migration method, characterized by, The method comprises: obtaining M to-be-migrated tables contained in a target transmission task in a task list, and obtaining a data filtering strategy of a database to which the M to-be-migrated tables belong, wherein M is a positive integer, and each to-be-migrated table contains data; removing part of the data in each to-be-migrated table in sequence according to the data filtering strategy to obtain M candidate tables, and generating a first file according to the data in each candidate table in sequence to obtain P first files, wherein P is a positive integer; determining a task type of the target transmission task, wherein the task type comprises a temporary task and a timing task; in a case where the task type is the timing task, sequentially sending each first file to a target storage table of the first file in a target data lake; in a case where the task type is the temporary task, obtaining a file name of each first file, merging the first files with the same file name to obtain N second files, and sequentially sending each second file to the target storage table of the first file in the target data lake, wherein N is a positive integer; The method further comprises: detecting whether a target table that has undergone table structure change exists in a plurality of initial tables stored in the database; in a case where the target table exists, obtaining table structure change content of the target table, sending the table structure change content to the target data lake, and recording a first preset time in the target table, wherein the table structure of a target storage table corresponding to the target table is changed according to the table structure change content, and the first preset time represents the latest change time of the target storage table corresponding to the target table.

2. The method of claim 1, wherein, In a case where the task type is the timing task, before the step of generating a first file according to the data in each candidate table in sequence to obtain P first files, the method further comprises: obtaining a preset file generation time of the target transmission task, and obtaining a current time, wherein the current time is the time at which the step of executing the M to-be-migrated tables contained in the target transmission task in the task list is performed; determining whether the preset file generation time and the current time are the same; in a case where the preset file generation time and the current time are the same, performing the step of generating a first file according to the data in each candidate table in sequence to obtain P first files; in a case where the preset file generation time and the current time are not the same, pausing the execution of the target transmission task until the current time and the preset file generation time are the same.

3. The method of claim 1, wherein, After the step of generating a first file according to the data in each candidate table in sequence to obtain P first files, the method further comprises: sequentially verifying each first file, and determining whether an abnormal file exists in the P first files according to a verification result; In the case that the abnormal file does not exist in the P first files, the generation result of each file is recorded in a file execution detail table, and a step of determining a task type of a transmission task of the M tables to be migrated is performed; In the case that the abnormal file exists in the P first files, the generation result of each file is recorded in the file execution detail table, the abnormal file is sent to a server, and the step of determining the task type of the target transmission task is performed according to the remaining non-abnormal first files.

4. The method of claim 1, wherein, After obtaining the table structure change content of the target table, the method further comprises: obtaining filtering content related to the target table from the data filtering strategy, and judging whether the table structure change of the target table affects the filtering content, wherein the filtering content indicates data content used to generate the first file; in the case that the table structure change does not affect the filtering content, canceling the step of sending the table structure change content to the target data lake, and recording the first preset time in the target table.

5. The method of claim 1, wherein, In the case that the task type is a timing task, P first files are generated by sequentially generating a first file according to data in each candidate table, including: For each candidate table, judging whether the table structure of the first table to be migrated corresponding to the candidate table changes; in the case that the table structure changes, judging whether the current time is earlier than the second preset time recorded in the candidate table, wherein the current time is the time of performing the step of obtaining the M tables to be migrated included in the target transmission task in the task list, and the second preset time represents the latest change time of the target storage table corresponding to the candidate table; in the case that the current time is not earlier than the second preset time, generating a first file according to the data in the candidate table; in the case that the current time is earlier than the second preset time, obtaining the change content of the first table to be migrated, and restoring the first table to be migrated according to the change content to obtain a second table to be migrated; re-generate the candidate table through the second table to be migrated, and generate a first file according to the data in the re-generated candidate table.

6. The method of claim 5, wherein, In the case that the table structure changes, the method further comprises: obtaining filtering content related to the candidate table from the data filtering strategy, and judging whether the table structure change of the candidate table affects the filtering content, wherein the filtering content indicates data content used to generate the first file; in the case that the table structure change does not affect the filtering content, performing the step of generating a first file according to the data in the candidate table.

7. A data migration apparatus, characterized by comprising: comprises: a first obtaining unit, configured to obtain M tables to be migrated included in a target transmission task in a task list, and obtain a data filtering strategy of a database to which the M tables to be migrated belong from the database, wherein M is a positive integer, and each table to be migrated contains data; The removing unit is configured to remove part of data in each table to be migrated according to the data screening strategy to obtain M candidate tables, and generate a first file according to data in each candidate table to obtain P first files; The determining unit is configured to determine a task type of the target transmission task, wherein the task type includes a temporary task and a timing task; The first sending unit is configured to send each first file to a target storage table of the first file in the target data lake in a case where the task type is the timing task. The second sending unit is configured to obtain a file name of each first file and merge first files with the same file name to obtain N second files in a case where the task type is the temporary task, and send each second file to the target storage table of the first file in the target data lake, wherein N is a positive integer. The apparatus further includes a detecting unit configured to detect whether a target table that has undergone table structure change exists in the plurality of initial tables stored in the database; and a third obtaining unit configured to, in a case where the target table exists, obtain table structure change content of the target table, send the table structure change content to the target data lake, and record a first preset time in the target table, and determine whether to generate the first file according to the current table structure of the target table according to the first preset time when generating the file according to the target table, wherein a table structure of a target storage table corresponding to the target table is changed according to the table structure change content, and the first preset time represents a latest change time of the target storage table corresponding to the target table.

8. A computer storage medium, characterized in that The computer storage medium is configured to store a program, wherein the program controls a device where the computer storage medium is located to execute the data migration method in any one of claims 1 to 6 when the program is executed.

9. An electronic device, comprising: The apparatus includes one or more processors and a memory, and the memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data migration method in any one of claims 1 to 6.

Images