Table change processing method and system

Abstract

The present invention provides a table change processing method and system, the method comprising: based on a data collection task, obtaining first table structure data information and second table structure data information; if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information; according to the table change trigger information, changing the table structure data information in the target end database, obtaining third table structure data information; based on the third table structure data information, collecting data in the source end database into the target end database. The present invention compares the table structure data information of the current source end database with the table structure data information of the source end database during the last data collection task, and then changes the table structure of the target database, avoiding manual changes to the downstream data structure and improving data collection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a table change processing method and system. Background Art

[0002] At present, information technology is in the era of big data technology development, and more and more large and medium-sized enterprises are building their own big data platforms. The first step in building a big data platform is to realize data concentration and form your own data lake or data warehouse. Therefore, enterprises generally have their own data collection tools or platforms to support the entry of external data into the lake and warehouse.

[0003] Existing data warehouse technology mainly utilizes ETL (Extract Transform Load) design implementation, namely data extraction, data cleaning and transformation, and data loading. Although existing ETL tools, such as dataX, FLinkX, and Kettle, are already very mature and have greatly reduced the development workload of data integration, there is still an unavoidable problem: when the data structure of the source database system changes, these tools or platforms cannot automatically update the table data structure on the target side, as well as the original data collection task information. However, most companies still rely on manual methods to update the table data structure on the target side and change and restart the original data collection task to address the above issues, resulting in reduced data collection efficiency.

[0004] Therefore, there is an urgent need for a table change processing method and system to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a table change processing method and system.

[0006] The present invention provides a table change processing method, comprising:

[0007] Based on the data collection task, first table structure data information and second table structure data information are obtained, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the last data collection;

[0008] If there is a difference in the version baselines between the first table structure data information and the second table structure data information, generating corresponding table change trigger information;

[0009] According to the table change trigger information, the table structure data information in the target database is changed to obtain the third table structure data information;

[0010] Based on the third table structure data information, the data in the source database is collected into the target database.

[0011] According to a table change processing method provided by the present invention, the table structure data information in the target database is changed according to the table change trigger information to obtain the third table structure data information, including:

[0012] Determine a table structure change method according to the table change trigger information;

[0013] By using the table structure change method and the data collection task, the corresponding table structure in the target database is changed, and based on the changed table structure, the table structure mapping relationship is updated to obtain an updated table structure mapping relationship;

[0014] According to the mapping relationship between the changed table structure and the updated table structure, third table structure data information is obtained.

[0015] According to a table change processing method provided by the present invention, after changing the table structure data information in the target database according to the table change trigger information and acquiring the third table structure data information, the method further includes:

[0016] Based on the updated table structure mapping relationship, the first field mapping relationship in the data collection task is updated to obtain a second field mapping relationship, and the first field mapping relationship is retained;

[0017] The first field mapping relationship is the field mapping relationship before the update in the data collection task; the second field mapping relationship is the field mapping relationship after the update in the data collection task;

[0018] The collecting the data in the source database into the target database based on the third table structure data information includes:

[0019] Based on the third table structure data information and the second field mapping relationship, the data in the source database is collected into the target database.

[0020] According to a table change processing method provided by the present invention, the method further includes:

[0021] Get table change failure information;

[0022] If the table change failure information is information about a table structure change failure, then re-changing the corresponding table structure in the target end database through the table structure change method and the data collection task;

[0023] If the table change failure information is information about a field mapping relationship change failure, the first field mapping relationship in the data collection task is updated again.

[0024] According to a table change processing method provided by the present invention, if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information includes:

[0025] Generate a first MD5 value and a second MD5 value respectively by using the MD5 algorithm, wherein the first MD5 value is the MD5 value corresponding to the first table structure data information, and the second MD5 value is the MD5 value corresponding to the second table structure data information;

[0026] If the first MD5 value and the second MD5 value are different, corresponding table change trigger information is generated.

[0027] According to a table change processing method provided by the present invention, the method of obtaining first table structure data information and second table structure data information based on a data acquisition task includes:

[0028] According to the data collection task, determine the source data source information and data collection date;

[0029] Determine a source database according to the source data source information;

[0030] The first table structure data information and the second table structure data information are obtained from the source database according to the data collection date.

[0031] The present invention also provides a table change processing system, comprising:

[0032] A data collection task processing module, used to obtain first table structure data information and second table structure data information based on the data collection task, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the last data collection;

[0033] A table change identification module, configured to generate corresponding table change trigger information if there is a difference in the version baseline between the first table structure data information and the second table structure data information;

[0034] A table change processing flow control module is used to change the table structure data information in the target end database according to the table change trigger information, and obtain the third table structure data information;

[0035] A data collection execution module is used to collect the data in the source-end database into the target-end database based on the third table structure data information.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the table change processing methods described above is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the table change processing method described in any one of the above is implemented.

[0038] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the table change processing method described above is implemented.

[0039] The present invention provides a table change processing method and system, which compares the table structure data information of the source database during the current data collection task with the table structure data information of the source database during the previous data collection task, determines the change result of the upstream data structure, and then automatically changes the table structure data information of the target database, thereby avoiding manual changes to the downstream data structure and improving data collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flow chart of a table change processing method provided by the present invention;

[0042] Figure 2 An overall schematic diagram of the table change processing flow provided by the present invention;

[0043] Figure 3 A schematic diagram of the structure of the table change processing system provided by the present invention;

[0044] Figure 4 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In existing technologies, such as Apache Griffin, as an open source big data quality solution, it supports two data quality detection methods: batch processing and streaming mode. It can measure data assets from different dimensions (for example, after the offline task is completed, check whether the data quantity in the source database and the target database is consistent, the number of null values ​​in the source table, etc.), thereby improving the accuracy and credibility of the data; Apache Atlas provides Hadoop clusters with core metadata governance capabilities including data classification, centralized policy engine, data lineage, security and lifecycle management, supports metadata management of Hive, Storm, Kafka, Hbase and Sqoop, and displays data lineage in the form of a gallery.

[0047] Although the existing technologies provide more powerful identification and analysis functions for table changes and support a wider range of database types, the actual learning, installation and maintenance costs are too high. These existing technologies are more inclined to the analysis and management of data assets, data quality and data lineage, and are not suitable for data collection scenarios. In addition, the existing technologies can only complete the identification and analysis of table changes, and cannot perform subsequent processing on the change details, let alone complete the automatic table change processing related to data collection. In particular, when the table structure of a table on the upstream end (i.e., the source database) changes, the data collection function for the table will not be able to guarantee data integrity and consistency, resulting in the inability to extract the upstream data to the downstream end (i.e., the target database).

[0048] The present invention aims at the data collection process. Before the collection, the table data structure of the target end database is automatically updated. At the same time, the data collection task information is automatically updated to ensure that the upstream end data structure and the downstream end data structure are completely consistent during each data collection. No manual intervention is required in the middle, and the user is unaware, thereby improving the data collection efficiency.

[0049] Figure 1 A flow chart of the table change processing method provided by the present invention is shown as follows: Figure 1 As shown, the present invention provides a table change processing method, comprising:

[0050] Step 101, based on the data collection task, obtain first table structure data information and second table structure data information, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the time of the last data collection.

[0051] In the present invention, firstly, the obtained data collection task is verified and analyzed for relevant parameters. Specifically, based on the above embodiment, the first table structure data information and the second table structure data information are obtained based on the data collection task, including:

[0052] According to the data collection task, determine the source data source information and data collection date;

[0053] Determine a source database according to the source data source information;

[0054] The first table structure data information and the second table structure data information are obtained from the source database according to the data collection date.

[0055] In the present invention, in response to the user's operation input, a corresponding data collection task is generated, and then the source data source information of this data collection task is obtained by parsing the data collection task, so as to determine the source database related information (for example, database address) according to the information; at the same time, for the collection date of this data collection task, in addition to obtaining the current latest table structure data information in the source database (i.e., the first table structure data information, including the table structure and mapping relationship, etc.) from the source database, the table structure data information in the source database at the time of the last data collection (i.e., the second table structure data information) is obtained through the most recent data collection task of the source database. In the present invention, when parsing the data collection task, in addition to obtaining the source data source information and the data collection date, the target data source information (the information includes the target database related information, for example, the table structure and address information of the target database, etc.) and the data collection method can also be obtained; at the same time, a unified database connection pool and thread pool are implemented to call up subsequent processing in a multi-threaded manner.

[0056] Step 102: If there is a difference in the version baseline between the first table structure data information and the second table structure data information, generate corresponding table change trigger information.

[0057] In the present invention, the data table structure information of the source table is obtained from the source database to form the latest version baseline of the table, and then compared with the version baseline of the source table in the source database in the previous data collection task. If the two version baselines are determined to be inconsistent during the comparison process, it is necessary to perform a table change operation on the target database, so as to generate corresponding trigger information according to the comparison result between the two version baselines.

[0058] Step 103: According to the table change trigger information, the table structure data information in the target database is changed to obtain third table structure data information.

[0059] In the present invention, after determining that the table structure of the target database needs to be changed, the change details are parsed, and the table structure data information of the two versions is compared field by field to generate the change details; further, according to the change details, the target table structure in the target database is changed, and, according to the new table structure mapping relationship in the target table structure, the field mapping relationship in the data acquisition task is updated to obtain the updated data acquisition task.

[0060] Step 104: Based on the third table structure data information, collect the data in the source database into the target database.

[0061] In the present invention, an external data collection task can be started by calling an interface, or a data collection process can be directly called by a command, or a third-party data collection tool can be directly called to implement it. The present invention triggers the updated data collection task based on the changed table structure of the target database according to the interface calling method, and realizes the extraction of data from the source database to the target database.

[0062] The present invention provides a table change processing method, which compares the table structure data information of the source database at the time of the current data collection task with the table structure data information of the source database at the time of the previous data collection task, determines the change result of the upstream data structure, and then automatically changes the table structure data information of the target database, thereby avoiding the manual change of the downstream data structure and improving the data collection efficiency.

[0063] Based on the above embodiment, the method of changing the table structure data information in the target database according to the table change trigger information to obtain the third table structure data information includes:

[0064] The table structure change mode is determined according to the table change trigger information.

[0065] In the present invention, after determining that there is a difference in the version baseline between the first table structure data information and the second table structure data information, the table change trigger information is used to trigger a field-by-field comparison of the table structure data information between the two versions, thereby determining the table structure change mode to generate corresponding table change details. For example, there is a field in the first table structure data information, but there is no such field in the second table structure data information, which means that the source database has added the field after the last data collection. Therefore, for the table structure data information in the target database, the field also needs to be added this time. Therefore, it can be determined that the table change mode this time is a newly added field, and the table change details are determined based on the specific field content. For example, the newly added field is user age, and the table change details this time are the newly added user age field.

[0066] The corresponding table structure in the target database is changed through the table structure change method and the data collection task, and the table structure mapping relationship is updated based on the changed table structure to obtain an updated table structure mapping relationship.

[0067] In the present invention, after determining the table structure change mode (such as the newly added fields in the above embodiment), based on the data acquisition task, determine the table structure in the target database involved in this table change, that is, a table in the source database has a newly added field, and in the downstream target database, determine which tables need to add the field. Since the data acquisition task includes the source data source information and the target data source information, the task can be parsed to determine the association between the tables, and then the tables in the target database can be changed according to the table structure change mode. Further, after completing the table structure change in the target database, the mapping relationship between the newly added fields in each table needs to be updated (such as creating a new mapping relationship, including the corresponding relationship between the table and the data file, and the relationship between the table field and the file field).

[0068] Based on the mapping relationship between the changed table structure and the updated table structure, third table structure data information is obtained. Based on the above embodiment, after the table structure data information in the target database is changed according to the table change trigger information and the third table structure data information is obtained, the method further includes:

[0069] Based on the updated table structure mapping relationship, the first field mapping relationship in the data collection task is updated to obtain a second field mapping relationship, and the first field mapping relationship is retained;

[0070] The first field mapping relationship is the field mapping relationship before updating in the data collection task; the second field mapping relationship is the field mapping relationship after updating in the data collection task.

[0071] In the present invention, after updating the table structure mapping relationship, the field mapping relationship of the data collection task also needs to be updated. For example, the original data collection task needs to collect student names and student ages, and a student ID field has been added to the latest table structure of the source database. By adding the student ID field to the table in the target end, a new table structure mapping relationship is established between the source and target ends. At the same time, the field mapping relationship in the data collection task also needs to be updated, that is, the new data collection task updates the mapping relationship of querying student ID information from the source database table with the newly added student ID field in the target database table.

[0072] The collecting the data in the source database into the target database based on the third table structure data information includes:

[0073] Based on the third table structure data information and the second field mapping relationship, the data in the source database is collected into the target database.

[0074] In the present invention, the version baseline of the table structure data information in the source database corresponding to the current data collection task is compared with the version information of the table structure data information in the source database at the time of the most recent data collection task. Once it is determined that the comparison result is inconsistent, the table change process is triggered, and then subsequent field-by-field comparison is performed to generate table change details.

[0075] Specifically, the first step of the table change processing flow is to update the table data structure in the target database. Since there are many choices for the target database for storing tables in an actual environment, based on the different table change details generated, the update operations of different target databases are not the same. The specific processing logic of the table change includes multiple change types, such as source field deletion, source field name modification, and source field type modification. The present invention uses the Hive database as the table structure update method of the target database to illustrate. During the table change process, it is ensured that the historical data of the target database is not affected, and the source data can be normally written into the target database according to the new data structure.

[0076] Specifically, for Hive database, updating the table structure will not delete the data. Therefore, the present invention determines the table structure change method according to the table change trigger information, obtains the table change details, and then uses the REPLACE COLUMNS syntax to redefine the table structure through the change table details, and updates the table structure mapping relationship corresponding to the changed table structure, thereby completing the table structure change of the target database, so as to obtain the third table structure data information according to the changed table structure and the updated table structure mapping relationship; then, update the field mapping relationship in the data collection task, and use the new table structure mapping relationship in the target database to overwrite the old mapping relationship, that is, update the first field mapping relationship in the data collection task to obtain the second field mapping relationship. Preferably, in the present invention, the old mapping relationship is backed up at the same time to perform version rollback; finally, based on the third table structure data information, execute the data collection task after the field mapping relationship is updated to complete the data extraction.

[0077] Based on the above embodiment, the method further includes:

[0078] Get table change failure information;

[0079] If the table change failure information is information about a table structure change failure, then re-changing the corresponding table structure in the target end database through the table structure change method and the data collection task;

[0080] If the table change failure information is information about a field mapping relationship change failure, the first field mapping relationship in the data collection task is updated again.

[0081] In the present invention, when a table change process fails, it is necessary to accurately obtain the location of the last failure based on the table change failure information. When the change is executed again, it can continue from the place where the last operation failed, thereby realizing re-running in various abnormal scenarios during the table change process.

[0082] Specifically, in the present invention, a table change processing flow function is added to ensure that when a change step fails to be processed, resulting in the change process not being completed normally, the steps and status at the time of failure are recorded. After the specific steps that caused the failure are determined and the problem is solved manually, the steps where the change failed can still be re-executed and subsequent processing can be continued, thereby achieving a highly fault-tolerant table change processing capability.

[0083] Based on the above embodiment, if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information includes:

[0084] Generate a first MD5 value and a second MD5 value respectively by using the MD5 algorithm, wherein the first MD5 value is the MD5 value corresponding to the first table structure data information, and the second MD5 value is the MD5 value corresponding to the second table structure data information;

[0085] If the first MD5 value and the second MD5 value are different, corresponding table change trigger information is generated.

[0086] The present invention integrates table change processing into the data collection process, and completes the table change processing of the target database without affecting the data collection performance. In the present invention, before the data collection task starts, the latest data structure of the relevant table is first obtained from the database of the upstream system (i.e., the source database); then, the latest data structure is compared with the data structure obtained by the most recent data collection task. Preferably, in order to reduce resource consumption and improve comparison performance, the present invention converts the table structure information into an MD5 value before comparison, rather than comparing each field in the table structure in turn. Only when it is determined that the two MD5 values ​​are inconsistent, it indicates that the table has changed, and it is necessary to compare field by field, and generate the table change details before performing table change processing.

[0087] The present invention adopts MD5 value comparison technology to realize the comparison between the current version and the historical version of the table structure in the source database, so that most data collection tasks without table changes are basically not affected, providing more efficient change identification capabilities.

[0088] Figure 2 The overall schematic diagram of the table change processing flow provided by the present invention can be referred to Figure 2 As shown, first, the data collection main program responds to the user's input, generates a corresponding data collection task, and then performs parameter analysis on the data collection task to determine the source database, collection time, and target database of this data collection task;

[0089] Then, after connecting to the source database, obtain the current source table data structure information and record it in the target database to form the latest version baseline; at the same time, compare it with the version of the source table data structure information in the previous collection task, and trigger subsequent instructions based on the comparison results. If they are inconsistent, proceed to the next step; if they are consistent, execute the current data collection task to complete data collection from the source to the target;

[0090] Next, when it is determined that the version baseline of the latest source table data structure is inconsistent with the version of the source table data structure information in the previous collection task, a table change detail is generated by comparing each field, and a corresponding change task record is created in the target database, and then according to the table change task, the target data structure change and data collection task information update are completed; in case of failure in the target data structure change and data collection task information update, different steps will be jumped to and processed again according to different states, that is, when the table structure change fails, the corresponding table structure in the target database is changed again through the table structure change method and data collection task; when the field mapping relationship change of the data collection task fails, the field mapping relationship in the data collection task is updated again;

[0091] Finally, after completing the change of the target end data structure and updating the data collection task information, the updated data collection task is called through the interface to synchronize the source end data to the target end.

[0092] The present invention proposes for the first time to integrate table change processing into the data collection process. The existing table change processing is independent of the data collection task, and an additional change verification function is added, resulting in no dependency between the two. There is a problem that when the change verification is triggered, the data collection task has already started to execute and the change operation cannot be performed. The table change processing flow is integrated into the data collection process, thereby serving as a prerequisite for the execution of the data collection task.

[0093] The table change processing system provided by the present invention is described below. The table change processing system described below and the table change processing method described above can be referenced to each other.

[0094] Figure 3 The structure diagram of the table change processing system provided by the present invention is as follows: Figure 3As shown, the present invention provides a table change processing system, including a data acquisition task processing module 301, a table change identification module 302, a table change processing flow control module 303 and a data acquisition execution module 304, wherein the data acquisition task processing module 301 is used to obtain a first table structure data information and a second table structure data information based on a data acquisition task, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the time of the last data acquisition; the table change identification module 302 is used to generate corresponding table change trigger information if there is a difference in the version baseline between the first table structure data information and the second table structure data information; the table change processing flow control module 303 is used to change the table structure data information in the target database according to the table change trigger information to obtain a third table structure data information; the data acquisition execution module 304 is used to collect the data in the source database into the target database based on the third table structure data information.

[0095] In the present invention, the acquisition task processing module 301 verifies and analyzes the relevant parameters of the acquired data acquisition task. Specifically, the acquisition task processing module 301 generates the corresponding data acquisition task in response to the user's operation input, and then obtains the source data source information of this data acquisition task by analyzing the data acquisition task, so as to determine the source database related information (for example, database address) based on the information; at the same time, the acquisition task processing module 301, in addition to obtaining the current latest table structure data information (i.e., the first table structure data information, including table structure and mapping relationship, etc.) in the source database from the source database for the acquisition date of this data acquisition task, also obtains the table structure data information (i.e., the second table structure data information) in the source database at the time of the last data acquisition through the most recent data acquisition task of the source database. In the present invention, when parsing the data collection task, the collection task processing module 301 can obtain not only the source data source information and the data collection date, but also the target data source information (this information includes relevant information of the target database, for example, the table structure and address information of the target database) and the data collection method, etc.; at the same time, a unified database connection pool and thread pool are implemented to call up subsequent processing in a multi-threaded manner.

[0096] Furthermore, the table change identification module 302 obtains the data table structure information of the source table from the source database, forms the latest version baseline of the table, and then compares it with the version baseline of the source table in the source database in the previous data collection task. If it is determined that the two version baselines are inconsistent during the comparison process, it is necessary to perform a table change operation on the target database, so as to generate corresponding trigger information according to the comparison result between the two version baselines.

[0097] Furthermore, after determining that the table structure of the target database needs to be changed, the table change processing flow control module 303 performs change detail analysis and compares the table structure data information of the two versions field by field to generate change details; then, the table change processing flow control module 303 performs change processing on the target table structure in the target database according to the change details, and updates the field mapping relationship in the data collection task according to the new table structure mapping relationship in the target table structure to obtain an updated data collection task.

[0098] Finally, the data collection execution module 304 triggers the updated data collection task based on the changed table structure of the target database through an interface call, thereby extracting data from the source database to the target database.

[0099] The present invention provides a table change processing system, which compares the table structure data information of the source database at the time of the current data collection task with the table structure data information of the source database at the time of the previous data collection task, determines the change result of the upstream data structure, and then automatically changes the table structure data information of the target database, thereby avoiding the manual change of the downstream data structure and improving the data collection efficiency.

[0100] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0101] Figure 4 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor (Processor) 401, a communication interface (Communications Interface) 402, a memory (Memory) 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 complete mutual communication through the communication bus 404. The processor 401 can call the logic instructions in the memory 403 to execute the table change processing method, which includes: based on the data acquisition task, obtaining the first table structure data information and the second table structure data information, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the last data acquisition; if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information; according to the table change trigger information, changing the table structure data information in the target database to obtain the third table structure data information; based on the third table structure data information, collecting the data in the source database into the target database.

[0102] In addition, the logic instructions in the above-mentioned memory 403 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0103] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the table change processing method provided by the above methods, and the method includes: based on a data acquisition task, obtaining first table structure data information and second table structure data information, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the time of the last data acquisition; if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generate corresponding table change trigger information; according to the table change trigger information, change the table structure data information in the target database to obtain third table structure data information; based on the third table structure data information, collect the data in the source database into the target database.

[0104] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the table change processing method provided in the above-mentioned embodiments, the method comprising: based on a data collection task, obtaining first table structure data information and second table structure data information, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the time of the last data collection; if there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information; according to the table change trigger information, changing the table structure data information in the target database to obtain third table structure data information; based on the third table structure data information, collecting the data in the source database into the target database.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A table change processing method, characterized in that: include: Based on the data collection task, first table structure data information and second table structure data information are obtained, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the last data collection; If there is a difference in the version baselines between the first table structure data information and the second table structure data information, generating corresponding table change trigger information; According to the table change trigger information, the table structure data information in the target database is changed to obtain the third table structure data information; Based on the third table structure data information, collecting the data in the source database into the target database; The step of obtaining the first table structure data information and the second table structure data information based on the data collection task includes: According to the data collection task, determine the source data source information and data collection date; Determine a source database according to the source data source information; Acquire first table structure data information and second table structure data information from the source database according to the data collection date; The step of changing the table structure data information in the target database according to the table change trigger information to obtain the third table structure data information includes: Determine a table structure change method according to the table change trigger information; By using the table structure change method and the data collection task, the corresponding table structure in the target database is changed, and based on the changed table structure, the table structure mapping relationship is updated to obtain an updated table structure mapping relationship; Acquire third table structure data information according to the mapping relationship between the changed table structure and the updated table structure; After changing the table structure data information in the target database according to the table change trigger information and acquiring the third table structure data information, the method further includes: Based on the updated table structure mapping relationship, the first field mapping relationship in the data collection task is updated to obtain a second field mapping relationship, and the first field mapping relationship is retained; The first field mapping relationship is the field mapping relationship before the update in the data collection task; the second field mapping relationship is the field mapping relationship after the update in the data collection task; The collecting the data in the source database into the target database based on the third table structure data information includes: Based on the third table structure data information and the second field mapping relationship, the data in the source database is collected into the target database.

2. The table change processing method according to claim 1, characterized in that: The method further comprises: Get table change failure information; If the table change failure information is information about a table structure change failure, then re-changing the corresponding table structure in the target end database through the table structure change method and the data collection task; If the table change failure information is information about a field mapping relationship change failure, the first field mapping relationship in the data collection task is updated again.

3. The table change processing method according to claim 1, characterized in that: If there is a difference in the version baseline between the first table structure data information and the second table structure data information, generating corresponding table change trigger information includes: Generate a first MD5 value and a second MD5 value respectively by using the MD5 algorithm, wherein the first MD5 value is the MD5 value corresponding to the first table structure data information, and the second MD5 value is the MD5 value corresponding to the second table structure data information; If the first MD5 value and the second MD5 value are different, corresponding table change trigger information is generated.

4. A table change processing system, characterized in that: include: A data collection task processing module, used to obtain first table structure data information and second table structure data information based on the data collection task, wherein the first table structure data information is the corresponding table structure data information in the source database at the current moment; the second table structure data information is the corresponding table structure data information in the source database at the last data collection; A table change identification module, configured to generate corresponding table change trigger information if there is a difference in the version baseline between the first table structure data information and the second table structure data information; A table change processing flow control module is used to change the table structure data information in the target end database according to the table change trigger information, and obtain the third table structure data information; A data collection execution module, used for collecting the data in the source database into the target database based on the third table structure data information; The data acquisition task processing module is specifically used for: According to the data collection task, determine the source data source information and data collection date; Determine a source database according to the source data source information; Acquire first table structure data information and second table structure data information from the source database according to the data collection date; The table change processing flow control module is specifically used for: Determine a table structure change method according to the table change trigger information; By using the table structure change method and the data collection task, the corresponding table structure in the target database is changed, and based on the changed table structure, the table structure mapping relationship is updated to obtain an updated table structure mapping relationship; Acquire third table structure data information according to the mapping relationship between the changed table structure and the updated table structure; The system is also used to: Based on the updated table structure mapping relationship, the first field mapping relationship in the data collection task is updated to obtain a second field mapping relationship, and the first field mapping relationship is retained; The first field mapping relationship is the field mapping relationship before the update in the data collection task; the second field mapping relationship is the field mapping relationship after the update in the data collection task; The collecting the data in the source database into the target database based on the third table structure data information includes: Based on the third table structure data information and the second field mapping relationship, the data in the source database is collected into the target database.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the table change processing method according to any one of claims 1 to 3 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the table change processing method according to any one of claims 1 to 3 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the table change processing method according to any one of claims 1 to 3 is implemented.

Images