Data processing method and device, equipment and storage medium

Abstract

The present disclosure provides a data processing method, device and equipment and storage medium, which can be applied to the field of computer and database. The method comprises: for a target application resource, extracting source table change data of N source tables respectively to obtain a source table change data set, wherein the N source tables are stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1; generating a source table change message according to the source table change data set, wherein the source table change message is sent to a message queue; parsing the source table change message obtained from the message queue to obtain the source table change data of the N source tables respectively; and migrating the source table change data of the N source tables respectively to a temporary table corresponding to the source table according to the correspondence between the temporary table and the source table to obtain a target temporary table corresponding to the N source tables respectively, wherein the target temporary table and the source table have different table structures.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer and database, and particularly relates to a data processing method and device, equipment, medium and program product. BACKGROUND

[0002] With the rapid development of technology, more and more enterprises meet the growing demand of business by building business service application resources such as mobile payment application, and improve the efficiency of business services. In related actual application scenarios, various types of databases can be used to support the operation of business service resources. However, during the upgrade of the database table structure, the business service resources usually need to be shut down for a long time to meet the table structure upgrade requirements, which has a great negative impact on the stable operation of the business service resources, and it is difficult to meet the actual business service requirements. SUMMARY

[0003] In view of the above problems, the present disclosure provides a data processing method, device, equipment, medium and program product.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0005] For a target application resource, extract source table change data of each of N source tables to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1;

[0006] According to the source table change data set, generate a source table change message, wherein the source table change message is sent to a message queue;

[0007] Parse the source table change message obtained from the message queue to obtain the source table change data of each of the N source tables; and

[0008] According to the correspondence between the temporary table and the source table, migrate the source table change data of each of the N source tables to the temporary table corresponding to the source table to obtain the target temporary table corresponding to each of the N source tables, wherein the target temporary table has a different table structure from the source table.

[0009] According to an embodiment of the present disclosure, the data processing method further comprises:

[0010] In the case of reaching the current mirroring time, obtain the source table change message generated in the first historical time period from the message queue;

[0011] Wherein, the first historical time period includes a time period formed by the current mirroring time and the last mirroring time adjacent to the current mirroring time.

[0012] According to an embodiment of the present disclosure, the migrating the N source table change data of each of the source tables to the temporary table corresponding to the source table according to the correspondence between the temporary table and the source table comprises:

[0013] determining the temporary table corresponding to the source table change data according to the source table identifier of the source table change data; and

[0014] migrating the source table change data to the temporary table based on a change data migration rule.

[0015] According to an embodiment of the present disclosure, the extracting the N source table change data of each of the source tables for the target application resource comprises:

[0016] obtaining M log files of each of the target databases; and

[0017] parsing the M log files corresponding to each of the target databases to obtain the N source table change data of each of the source tables.

[0018] According to an embodiment of the present disclosure, the data processing method further comprises:

[0019] determining the target table structure corresponding to each of the N source tables according to the table structure change instruction; and

[0020] creating the temporary table corresponding to each of the N source tables in the M target databases according to the target table structure corresponding to each of the N source tables.

[0021] According to an embodiment of the present disclosure, the source table change data contains a target application resource identifier of the target application resource;

[0022] The generating the source table change message according to the source table change data set comprises:

[0023] generating a message index corresponding to the source table change data set according to the target application resource identifier; and

[0024] encapsulating the message index and the source table change data set based on a preset protocol to obtain the source table change message.

[0025] According to an embodiment of the present disclosure, the data processing method further comprises:

[0026] in the case that the target application resource stops running, updating a target temporary table name of the target temporary table to a source table name of a source table corresponding to the target temporary table.

[0027] According to an embodiment of the present disclosure, the data processing method further comprises:

[0028] The time attribute information of the M target databases is unified.

[0029] According to an embodiment of the present disclosure, the message queue comprises at least one of:

[0030] A Kafka message queue and a Rocket message queue.

[0031] Another aspect of the present disclosure further provides a data processing apparatus, comprising:

[0032] A source table change data extraction module configured to extract source table change data of N source tables respectively for a target application resource to obtain a source table change data set, wherein the N source tables are stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1.

[0033] A source table change message generation module configured to generate source table change messages according to the source table change data set, wherein the source table change messages are sent to a message queue.

[0034] An analysis module configured to analyze the source table change messages obtained from the message queue to obtain source table change data of the N source tables respectively.

[0035] A migration module configured to migrate the source table change data of the N source tables respectively to temporary tables corresponding to the source tables according to a corresponding relationship between the temporary tables and the source tables to obtain target temporary tables corresponding to the N source tables respectively, wherein the target temporary tables have different table structures from the source tables.

[0036] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above method.

[0037] Another aspect of the present disclosure further provides a computer readable storage medium having stored executable instructions, which are executed by a processor to cause the processor to perform the above method.

[0038] Another aspect of the present disclosure further provides a computer program product comprising a computer program, which is executed by a processor to implement the above method.

[0039] According to the data processing method, device, equipment, medium and program product provided by the present disclosure, the source table change data with different database formats can be unified into the structured storage data of the message queue by generating the source table change message with the same data structure according to the source table change data extracted from the multiple databases, and the source table change data can be conveniently migrated to the temporary table with different table structures corresponding to the source table by analyzing the source table change message obtained from the message queue, and the table structure upgrade of the source table of the multiple types of databases is realized, so that the technical problem of long database downtime in the table structure upgrade process is at least partially solved, and the technical effect of improving the database operation efficiency is realized. BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0041] Figure 1 An application scenario diagram of the data processing method and device according to an embodiment of the present disclosure is schematically shown;

[0042] Figure 2 A flowchart of the data processing method according to an embodiment of the present disclosure is schematically shown;

[0043] Figure 3 A flowchart of the data processing method according to another embodiment of the present disclosure is schematically shown;

[0044] Figure 4 A flowchart of generating the source table change message according to the source table change data set according to an embodiment of the present disclosure is schematically shown;

[0045] Figure 5 An application scenario diagram of the data processing method according to an embodiment of the present disclosure is schematically shown;

[0046] Figure 6 A structural block diagram of the data processing device according to an embodiment of the present disclosure is schematically shown;

[0047] Figure 7 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments, or one or more embodiments, can be practiced without these specific details. In other instances, well-known structures and

[0049] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," means the inclusion of but not limited to, and is not meant to be construed as a functionally limiting the scope of the embodiments.

[0050] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings that are consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0051] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any of one, all, or a combination thereof. For example, "a system having at least one of A, B, and C" should be interpreted to include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.

[0052] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (such as including but not limited to user personal information) comply with relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0053] In the process of implementing the concept of the present disclosure, the inventors found that in the case of table structure upgrade of a database for the same business application resource, a plurality of database types are usually involved, such as MySQL, ORACLE, and other databases, which need to be upgraded. In the related art, only the table structure of a plurality of databases belonging to the same business application resource can be upgraded respectively, and it is difficult to complete the upgrade work for the business application resource in the same coordination, resulting in a long downtime of the business application resource and affecting the quality of service.

[0054] Embodiments of the present disclosure provide a data processing method, apparatus, device and storage medium. The data processing method comprises: for a target application resource, extracting source table change data of N source tables respectively to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1; generating a source table change message according to the source table change data set, wherein the source table change message is sent to a message queue; parsing the source table change message obtained from the message queue to obtain source table change data of the N source tables respectively; and migrating the source table change data of the N source tables respectively to a temporary table corresponding to the source table according to a correspondence between the temporary table and the source table to obtain a target temporary table corresponding to the N source tables respectively, wherein the target temporary table and the source table have different table structures.

[0055] According to embodiments of the present disclosure, by generating a source table change message with the same data structure according to the source table change data extracted from the plurality of databases, the source table change data with different database formats can be unified as structured storage data of the message queue, and meanwhile, parsing the source table change message obtained from the message queue can conveniently migrate the source table change data to the temporary table corresponding to the source table and having different table structures, thereby realizing table structure upgrade of the source tables of the plurality of types of databases in parallel, reducing the operation time length and the database downtime of the table structure upgrade, and improving the database operation efficiency.

[0056] Figure 1 An application scenario diagram of the data processing method and apparatus according to embodiments of the present disclosure is schematically shown.

[0057] As shown in Figure 1 The application scenario 100 according to the embodiments can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105 and a message queue server 106. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0058] A user can use at least one of the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0059] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0060] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0061] The message queue server 106 may include a distributed message queue server or server cluster in related technologies, such as the Kafka message queue server.

[0062] It should be noted that the data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0063] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0064] The following will be based on Figure 1 The described scene, through Figures 2 to 5 The data processing method of the disclosed embodiments will be described in detail.

[0065] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0066] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S240.

[0067] At operation S210, source table change data of N source tables is extracted for a target application resource, to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1.

[0068] According to an embodiment of the present disclosure, the target application resource can be an application resource in an application system for implementing a business service function, for example, a transaction application resource capable of implementing a transaction service function, and the like. The target application resource can perform a database operation such as deletion and insertion on the source tables from the M different target databases. Correspondingly, the N source tables can generate corresponding source table change data based on the database operation of the target application resource.

[0069] At operation S220, a source table change message is generated according to the source table change data set, wherein the source table change message is sent to a message queue.

[0070] At operation S230, the source table change message obtained from the message queue is parsed to obtain source table change data of the N source tables.

[0071] At operation S240, the source table change data of the N source tables is migrated to the temporary table corresponding to the source table according to the correspondence between the temporary table and the source table, to obtain a target temporary table corresponding to the N source tables, wherein the target temporary table has a different table structure from the source table.

[0072] According to an embodiment of the present disclosure, the message queue can be a server or a server cluster capable of implementing message distributed storage and message interaction. The source table change data set can be encapsulated based on a preset communication protocol to generate a source table change message, and the message queue can store the source table change message for consumption by the target application resource or other application resources and application systems.

[0073] According to an embodiment of the present disclosure, the correspondence between the temporary table and the corresponding source table can be established through an index value, so that the source table change data generated in the source table is migrated to the temporary table corresponding to the source table, so that the target temporary table after migration can implement table structure upgrade for the source table, while not affecting the database operation of the target database corresponding to the source table, avoiding long-time shutdown of the database caused by database table structure change.

[0074] According to an embodiment of the present disclosure, by generating source table change messages with the same data structure from source table change data extracted from multiple databases, source table change data with different database formats can be unified into structured storage data of a message queue, and parsing source table change messages obtained from the message queue can conveniently migrate source table change data to temporary tables with different table structures corresponding to source tables, thereby realizing table structure upgrade of source tables of multiple types of databases in parallel, reducing operation time and database downtime of table structure upgrade, and improving database operation efficiency.

[0075] According to an embodiment of the present disclosure, the data processing method can further include the following operations.

[0076] In the case where the target application resource stops running, the target temporary table name of the target temporary table is updated to the source table name of the source table corresponding to the target temporary table.

[0077] According to an embodiment of the present disclosure, by updating the target temporary table name to the corresponding source table name, M target databases of different database types can simultaneously generate new source tables, that is, table structure upgrade of database tables (that is, source tables) in multiple target databases corresponding to the target application resource is realized. The data processing method provided by the embodiment of the present disclosure can realize database table structure upgrade operation across different database types, which can only occupy a short downtime of the target application resource, and complete the table structure upgrade operation that usually takes several hours in related technologies within a second, thereby improving the overall operation efficiency of the target application resource.

[0078] According to an embodiment of the present disclosure, the message queue can include a distributed message queue in related technologies, and by using the distributed storage, message sending and receiving performance of the distributed message queue, real-time structured storage of source table change data with different database types is realized, and high-throughput and high-performance transmission of table change messages is realized.

[0079] According to an embodiment of the present disclosure, the message queue includes at least one of the following: a Kafka message queue, a Rocket message queue (RocketMQ).

[0080] According to an embodiment of the present disclosure, the data processing method can further include the following operations.

[0081] The time attribute information of the M target databases is unified.

[0082] According to an embodiment of the present disclosure, the database clocks of the M target databases can be unified to avoid data time attribute errors in the process of migrating source table change data to temporary tables, thereby improving the database upgrade efficiency.

[0083] It should be understood that the time attribute information unification of the M target databases can be implemented by a time attribute information unification device such as a global clock module.

[0084] Figure 3 A flowchart of a data processing method according to another embodiment of the present disclosure is schematically shown.

[0085] As Figure 3 shown, the data processing method of this embodiment further includes operation S310 to operation S320.

[0086] In operation S310, the target table structure corresponding to each of the N source tables is determined according to the table structure change instruction.

[0087] In operation S320, a temporary table corresponding to each of the N source tables is respectively created in the M target databases according to the target table structure corresponding to each of the N source tables.

[0088] According to an embodiment of the present disclosure, the fields, field names, field types, widths, and other attributes of the temporary tables can constitute the target table structure,

[0089] According to an embodiment of the present disclosure, the created temporary tables can include empty tables having the target table structure. Alternatively, the created temporary tables can also store the inventory data in the respective corresponding source tables, as long as the temporary tables have the target table structure.

[0090] According to an embodiment of the present disclosure, in operation S210, extracting the source table change data of each of the N source tables for the target application resource can include the following operations.

[0091] Obtaining the log files of the M target databases, and parsing the log files corresponding to each of the M target databases to obtain the source table change data of each of the N source tables.

[0092] According to an embodiment of the present disclosure, by parsing the log files of each target database, the incremental data of each source table, i.e., the source table change data, can be obtained according to the operation records and other data change records obtained by parsing the log files, so that the source table change data can be obtained in time without affecting the running of the target application resource, thereby reducing the impact on the running of the target application resource.

[0093] According to an embodiment of the present disclosure, the data processing method can further include the following operations.

[0094] In the case of reaching the current mirroring time, the source table change messages generated in the first historical time period are obtained from the message queue.

[0095] The first historical time period includes a time period formed by the current mirroring time and the last mirroring time adjacent to the current mirroring time.

[0096] According to embodiments of this disclosure, the mirror moment can represent the moment when source table change data needs to be migrated to a temporary table. As the consumer of source table change messages, the target application resource can consume the generated source table change messages within a first historical time period comprised of adjacent mirror moments, thereby migrating the source table change data to the corresponding upgraded temporary table in a timely manner without affecting the operation of the target application resource. By setting appropriate mirror moments, batch updates of temporary tables can be achieved, improving the update efficiency of obtaining the target temporary table.

[0097] According to embodiments of this disclosure, by setting the mirroring time, the changed data of the source table is migrated to the corresponding temporary table in real time, thereby promoting the consistent mirroring of incremental data by iteratively generating the latest target temporary table, effectively ensuring the consistency of data between the source table and the temporary table at all times.

[0098] It should be noted that source table change messages obtained within a first historical time period can be parsed to improve the update frequency of the target temporary table. Alternatively, source table change messages can be obtained in batches after obtaining source table change messages corresponding to multiple first historical time periods. The embodiments of this disclosure do not limit the specific method of parsing source table change messages, and those skilled in the art can choose according to actual needs.

[0099] According to embodiments of this disclosure, the source table change dataset can be extracted from log files generated during a second historical time period. The second historical time period can be set based on the processor's processing power. Embodiments of this disclosure do not limit the specific setting method of the second historical time period.

[0100] According to embodiments of this disclosure, the source table change data packet contains the target application resource identifier of the target application resource.

[0101] Figure 4 The flowchart illustrating the generation of a source table change message based on a source table change dataset according to an embodiment of the present disclosure is shown schematically.

[0102] like Figure 4 As shown, in operation S220, source table change messages are generated based on the source table change dataset, including operations S410 to S420.

[0103] In operation S410, a message index corresponding to the source table change dataset is generated based on the target application resource identifier.

[0104] When operating S420, the message index and source table change dataset are encapsulated based on a preset protocol to obtain the source table change message.

[0105] According to an embodiment of the present disclosure, the source table change message can take the target application resource identifier as a message index stored in the message queue, so that in the case that the message queue stores a plurality of source table change messages generated by the target application resource, the different target application resources can obtain the source table change message from the message queue through the message index, thereby avoiding errors in the source table change message.

[0106] According to an embodiment of the present disclosure, the source table name, the temporary table name, the index field corresponding to the source table change data, and other attribute information can be collectively generated into the message index, so as to ensure the consistency of the source table change message stored in the message queue.

[0107] According to an embodiment of the present disclosure, in operation S240, according to the correspondence between the temporary table and the source table, the source table change data of the N source tables is migrated to the temporary table corresponding to the source table, which can include the following operations:

[0108] According to the source table identifier of the source table change data, the temporary table corresponding to the source table change data is determined, and the source table change data is migrated to the temporary table based on the change data migration rule.

[0109] According to an embodiment of the present disclosure, the source table name (source table identifier), the temporary table name (temporary table identifier), and the index field can also be used as the index of the corresponding temporary table associated with the source table change data, so as to realize the migration of the source table change data to the corresponding table position of the temporary table, avoid errors in the migration of the source table change data, and reduce the failure rate of the target resource.

[0110] Figure 5 An application scenario diagram of a data processing method according to an embodiment of the present disclosure is schematically shown.

[0111] As shown in Figure 5 The application scenario can include a target application resource 510 and a message queue 520. The target application resource 510 can be a business service application in a business service system, and the target application resource 510 can correspond to a target database 513 and a target database 514. The target database 513 and the target database 514 can be of different database types, for example, ORACLE database and MySQL database. The message queue 520 can include a distributed message engine, a distributed storage system, a message receiving module, a message sending module, and other message processing modules, and is used for communication and interaction with the target application resource 510.

[0112] The target database 513 can store source tables T11, T12, and T13, and the target database 514 can store source tables T21, T22, and T23. The source tables T11, T12, and T13, and the source tables T21, T22, and T23 can be source tables corresponding to the target application resource.

[0113] According to the target table structure obtained according to the table structure change instruction, the temporary tables T11', T12', and T13' corresponding to the source tables T11, T12, and T13 respectively, and the temporary tables T21', T22', and T23' corresponding to the source tables T21, T22, and T23 respectively can be generated. In this way, the source tables in the target database can be upgraded in table structure through the temporary tables T11', T12', T13', T21', T22', and T23'.

[0114] Before processing the incremental data, that is, before processing the source table change data, the source tables T21, T22, and T23, and the inventory data in the source tables T21, T22, and T23 can be batch migrated to the temporary tables T11', T12', T13', T21', T22', and T23' according to the migration rule, so as to upgrade the table structures of the inventory data tables of the target databases 513 and 514.

[0115] The log analysis unit 511 can analyze the log files generated by the target application resource 510 in the second historical time period in parallel through multiple links to obtain the source table change data of the source tables of the target databases 513 and 514 respectively. For example, binary codes can be extracted from the log files, and operation records (such as operation fields starting with UPDATE, DELETE, INSERT, etc.) for the source tables T11, T12, T13, T21, T22, and T23 can be obtained by analysis.

[0116] The source table change data corresponding to the plurality of source tables can be encapsulated according to the protocol to obtain source table change messages that can be identified by the message queue 520. The source table change messages can have a target application resource identifier corresponding to the target application resource 510, which can be used as a unique index (i.e., a message index) stored in the message queue 520 to avoid different application resources from obtaining the source table change messages. Correspondingly, the source table change data in the source table change messages can each have source table name identifiers and other index information, so as to facilitate migration of the source table change data to the temporary tables corresponding to the source tables according to the index information.

[0117] It should be noted that the plurality of source table change data in the source table change messages can also be spliced into a string in any structured data format such as xml format or Json format, so as to facilitate distributed storage of the message queue 520.

[0118] At the current mirroring time, the incremental data mirroring unit 512 can obtain the source table change messages generated in the first historical time period from the message queue 520. The source table change messages are parsed to obtain the source table change data corresponding to the temporary tables T11', T12', T13', T21', T22', and T23', respectively, for example, the operation records or query statements corresponding to each temporary table can be parsed, and then the source table change data is migrated to the corresponding temporary table according to the index information, thereby generating the target temporary tables T11', T12', T13', T21', T22', and T23' corresponding to the first historical time period.

[0119] In addition, if it is found during the parsing process that the related source table change message does not meet the parsing condition, it can be selected through the pre-defined configuration file whether to skip the abnormal source table change data, and the abnormal source table change data is stored in the exception table.

[0120] It should be noted that the multiple source table change messages in the first historical time period can be stored in the distributed storage module in time sequence in the message queue 520.

[0121] At the same time, the target databases 513 and 514 can be accessed through the database time sequence control node to obtain the time stamps of the log files of the target application resource 510 in which the source table change data extraction has been performed, and the latest historical time represented by the latest time stamp in the multiple time stamps is taken as the last mirroring time, so that the source table change data extraction is performed on all log files, and the time attributes of the target databases are unified.

[0122] By continuously determining the new mirroring time and the last mirroring time, the incremental data (i.e., the source table change data) of the target databases 513 and 514 can be iteratively obtained from the message queue 520, and the source table change data is migrated to the corresponding temporary table in real time according to the table structure requirements of the temporary table, thereby promoting the consistency mirroring of the incremental data by generating the latest target temporary table, and effectively guaranteeing the consistency of the source table and the temporary table data at each moment.

[0123] According to actual business requirements, the target application resource 510 can be stopped, that is, the target application resource 510 is stopped in a short time. Then, the source table names of the source tables T11, T12, T13, T21, T22, and T23 are updated to the target temporary table names of the latest target temporary tables, and the target temporary table names of the target temporary tables T11', T12', T13', T21', T22', and T23' are updated to the corresponding source table names, so as to realize the exchange of the table names between the source tables and the target temporary tables. In this way, the running of the target application resource 510 can be supported based on the latest target temporary tables T11', T12', T13', T21', T22', and T23', the database table structure upgrade operation across databases and across business application platforms is realized, and the database table structure upgrade operation can be completed in tens of seconds by using the data processing method provided in the embodiment of the present disclosure, so that the efficiency of the table structure upgrade is greatly improved, and the downtime of the target application resource during service stop is reduced.

[0124] Based on the above data processing method, the present disclosure further provides a data processing apparatus. The following will be described in detail Figure 6 The apparatus is described in detail.

[0125] Figure 6 The structure block diagram of the data processing apparatus according to the embodiment of the present disclosure is schematically shown.

[0126] As Figure 6 shown, the data processing apparatus 600 of the embodiment includes a source table change data extraction module 610, a source table change message generation module 620, an analysis module 630, and a migration module 640.

[0127] The source table change data extraction module 610 is configured to extract, for a target application resource, source table change data of N source tables respectively, to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1.

[0128] The source table change message generation module 620 is configured to generate a source table change message according to the source table change data set, wherein the source table change message is sent to a message queue.

[0129] The analysis module 630 is configured to analyze the source table change message obtained from the message queue to obtain the source table change data of the N source tables respectively.

[0130] The migration module 640 is configured to migrate, according to a correspondence between a temporary table and a source table, the source table change data of the N source tables respectively to the temporary table corresponding to the source table, to obtain a target temporary table corresponding to the N source tables respectively, wherein the target temporary table has a different table structure from the source table.

[0131] According to an embodiment of the present disclosure, the data processing apparatus further comprises a source table change message obtaining module.

[0132] The source table change message obtaining module is configured to, in a case where a current mirroring time point is reached, obtain, from the message queue, source table change messages generated in a first historical time period.

[0133] The first historical time period comprises a time period from the current mirroring time point to a previous mirroring time point adjacent to the current mirroring time point.

[0134] The migration module comprises a first determination unit and a migration unit.

[0135] The first determination unit is configured to determine, according to a source table identifier of the source table change data, a temporary table corresponding to the source table change data.

[0136] The migration unit is configured to migrate the source table change data to the temporary table based on a change data migration rule.

[0137] According to an embodiment of the present disclosure, the source table change data extraction module comprises a log file obtaining unit and a log file parsing unit.

[0138] The log file obtaining unit is configured to obtain log files of the M target databases respectively.

[0139] The log file parsing unit is configured to parse the log files corresponding to the M target databases respectively to obtain source table change data of the N source tables respectively.

[0140] According to an embodiment of the present disclosure, the data processing apparatus further comprises a target table structure determination module and a temporary table construction module.

[0141] The target table structure determination module is configured to determine target table structures corresponding to the N source tables respectively according to a table structure change instruction.

[0142] The temporary table construction module is configured to create temporary tables corresponding to the N source tables respectively in the M target databases according to the target table structures corresponding to the N source tables respectively.

[0143] According to an embodiment of the present disclosure, the source table change data comprises a target application resource identifier of a target application resource.

[0144] The source table change message generation module comprises a message index generation unit and a source table change message encapsulation unit.

[0145] The message index generation unit is configured to generate a message index corresponding to the source table change data set according to the target application resource identifier.

[0146] The source table change message encapsulation unit encapsulates the message index and the source table change data set based on a preset protocol to obtain a source table change message.

[0147] According to an embodiment of the present disclosure, the data processing apparatus further comprises an updating module.

[0148] The updating module is configured to, in a case where the target application resource stops running, update a target temporary table name of the target temporary table to a source table name of a source table corresponding to the target temporary table.

[0149] According to an embodiment of the present disclosure, the data processing apparatus further comprises a time attribute information unification module.

[0150] The time attribute information unification module is configured to unify time attribute information of the M target databases.

[0151] According to an embodiment of the present disclosure, the message queue comprises at least one of the following:

[0152] A Kafka message queue and a Rocket message queue.

[0153] According to an embodiment of the present disclosure, any multiple modules of the source table change data extraction module 610, the source table change message generation module 620, the parsing module 630 and the migration module 640 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the source table change data extraction module 610, the source table change message generation module 620, the parsing module 630 and the migration module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware and firmware implementation or in any appropriate combination of any of them. Alternatively, at least one of the source table change data extraction module 610, the source table change message generation module 620, the parsing module 630 and the migration module 640 can be at least partially implemented as a computer program module which, when executed, can perform the corresponding functions.

[0154] Figure 7 A block diagram of an electronic device suitable for implementing the data processing method according to an embodiment of the present disclosure is schematically shown.

[0155] As Figure 7As shown, the electronic device 700 according to embodiments of the present disclosure includes a processor 701 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage section 708. The processor 701 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0156] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 702 and / or the RAM 703. Note that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0157] According to embodiments of the present disclosure, the electronic device 700 can also include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read out therefrom is installed in the storage section 708 as necessary.

[0158] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to embodiments of the present disclosure.

[0159] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories such as the ROM 702 and / or the RAM 703 described above and / or one or more memory other than the ROM 702 and the RAM 703.

[0160] Embodiments of the present disclosure also include a computer program product that includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the present disclosure.

[0161] The above-described functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0162] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium and installed and downloaded through the communication part 709 and / or installed from the detachable medium 711. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0163] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709 and / or installed from the detachable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0164] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0165] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions readable by a machine (e.g., a computer). The instructions in the machine-readable medium focus on the function of the computer program (e.g., in the form of a program unit) and on the cooperation behavior between such program units. The program can be distributed over networks coupled to one or more processors or computers, and can be stored on computer readable media including computer program products, hard drives, RAMs, or any other device (internal or external) from which a processor or computer reads incoming data or instructions. In addition to being distributed over the Internet, or over any other publicly or privately accessed network, the aforementioned program product can also be stored in the memory of the respective machine and / or processor.

[0166] It will be appreciated by persons skilled in the art that features of various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such combinations or integrations are not expressly disclosed in the present disclosure. In particular, features of various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0167] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A data processing method comprising: extracting, for a target application resource, source table change data of N source tables respectively to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1; generating, according to the source table change data set, a source table change message, wherein the source table change message is sent to a message queue; parsing the source table change message obtained from the message queue to obtain source table change data of the N source tables respectively; and migrating, according to a correspondence between a temporary table and the source table, the source table change data of the N source tables respectively to a temporary table corresponding to the source table to obtain a target temporary table corresponding to the source table respectively, wherein the target temporary table has a different table structure from the source table; the method further comprising: in a case where the target application resource stops running, updating a target temporary table name of the target temporary table to a source table name of a source table corresponding to the target temporary table.

2. The method of claim 1, further comprising: in a case where a current mirroring time point is reached, obtaining, from the message queue, source table change messages generated in a first historical time period; wherein the first historical time period includes a time period formed by a current mirroring time point and a last mirroring time point adjacent to the current mirroring time point.

3. The method of claim 1, wherein, migrating, according to a correspondence between a temporary table and the source table, the source table change data of the N source tables respectively to a temporary table corresponding to the source table comprises: determining, according to a source table identifier of the source table change data, a temporary table corresponding to the source table change data; and migrating the source table change data to the temporary table based on a change data migration rule.

4. The method of claim 1, wherein, extracting, for a target application resource, source table change data of N source tables respectively comprises: obtaining log files of the M target databases respectively; and parsing the log files corresponding to the M target databases respectively to obtain the source table change data of the N source tables respectively.

5. The method of claim 1, further comprising: determining target table structures corresponding to the N source tables respectively according to a table structure change instruction; and creating, according to the target table structures corresponding to the N source tables respectively, the temporary tables corresponding to the N source tables respectively in the M target databases. the source table change data contains a target application resource identifier of the target application resource; 6. The method of claim 1, wherein, wherein generating, according to the source table change data set, a source table change message comprises: generating, according to the target application resource identifier, a message index corresponding to the source table change data set; and encapsulating the message index and the source table change data set based on a preset protocol to obtain the source table change message.

7. The method of any one of claims 1 to 6, further comprising: unifying time attribute information of the M target databases. the message queue comprises at least one of the following:

8. The method of claim 1, wherein, a Kafka message queue, a Rocket message queue.

9. A data processing apparatus comprising: ​ a source table change data extraction module configured to extract source table change data of each of N source tables for a target application resource to obtain a source table change data set, wherein the N source tables are respectively stored in M target databases corresponding to the target application resource, the M target databases have different database types, and M and N are positive integers greater than 1; a source table change message generation module configured to generate a source table change message according to the source table change data set, wherein the source table change message is sent to a message queue; a parsing module configured to parse the source table change message obtained from the message queue to obtain source table change data of each of the N source tables; and a migration module configured to migrate the source table change data of each of the N source tables to a temporary table corresponding to the source table according to a correspondence between the temporary table and the source table to obtain a target temporary table corresponding to each of the N source tables, wherein the target temporary table has a different table structure from the source table; an updating module configured to update a target temporary table name of the target temporary table to a source table name of a source table corresponding to the target temporary table in a case where the target application resource stops running. 10.An electronic device, comprising: one or more processors; a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method according to any one of claims 1-8. 11.A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-8. 12.A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images