Multi-table association method and device, equipment and storage medium

Abstract

The present disclosure relates to a multi-table association method and device, equipment and storage medium, belonging to the technical field of big data, the method comprises: analyzing the received multi-table association task, generating an execution chain for multi-table association; listen to each table in the multi-table association task, when the current table updates the log data, parse the log data updated by the current table into a message, and supplement the information of the parsed message to obtain the complete message of the current table; according to the execution chain of multi-table association, the complete message of the current table is associated to the table to be associated in the multi-table association task, and each node of the execution chain executes the message processing tasks such as listening, judging, parsing and association, which are all carried out in the memory, without the need of middleware and the process of serialization and deserialization, the message processing process occupies less disk space, and the disk consumption is low.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data, and particularly relates to a multi-table association method and device, equipment and a storage medium. BACKGROUND

[0002] In the logistics field, production data is generally stored using MySQL (MySQL is a relational database management system), and even for large data volume business, the mode adopted is still MySQL database and table splitting. In order to be used for business daily analysis and decision-making, it is often necessary to associate the data of multiple tables, and the mode adopted is to collect Binlog (Binlog is a binary format file used to record user SQL statement information for updating the database), and form a large wide table after real-time processing.

[0003] In the prior art, for example, Apache Flink (a stream data distributed processing engine) can realize real-time incremental data processing of a wide table composed of multiple single-table association calculations. For a MySQL type data source, there are mainly two ways to realize processing by Flink, one is to import production data Binlog into a message queue, and then process the content in the message queue by a Flink job, and the other is to directly collect Binlog for processing by Flink, without the need for a message queue. Since multiple table association needs to be performed, the messages generated first need to be cached, and then the data arriving later is associated, and the Flink production environment scheme is to store the data in RocksDB (a key-value storage system from Facebook that supports embedding and persistence). At the same time, Flink will be divided into different tasks according to the parallelism set by the job, and then these tasks will be allocated to different task managers in the cluster for running.

[0004] However, in the first mode, the production data Binlog is imported into a message queue for data processing, which introduces a new middleware (message queue); in the second mode, since the job tasks are allocated to different hosts in the Flink cluster for running, the message processing between different nodes increases the serialization and deserialization (serialization is the process of converting an object into a byte sequence, and deserialization is the process of converting a byte sequence into an object) process, which consumes a large amount of CPU resources. Moreover, multiple table data needs to be cached in RocksDB, which consumes a large amount of disk space, and also needs serialization and deserialization, which further increases the CPU consumption, making the process of associating multiple table data costly and time-consuming. SUMMARY

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a multi-table association method and device, equipment and storage medium.

[0006] In a first aspect, embodiments of the present disclosure provide a multi-table association method, the method comprising:

[0007] parsing the received multi-table association task to generate an execution chain for multi-table association, wherein the execution chain comprises a plurality of nodes connected in sequence;

[0008] listening to each table in the multi-table association task;

[0009] when the current table is updated with log data, parsing the log data updated by the current table into a message, and supplementing information of the parsed message to obtain a complete message of the current table;

[0010] associating the complete message of the current table to a table to be associated in the multi-table association task according to the execution chain of the multi-table association.

[0011] In a possible implementation, the method further comprises:

[0012] generating a corresponding execution chain for each table in the multi-table association task to obtain an execution chain for each table;

[0013] connecting the execution chain for each table according to the association relationship between the multi-tables to generate the execution chain for multi-table association.

[0014] In a possible implementation, before the log data updated by the current table is parsed into a message, the method further comprises:

[0015] when the number of log data updated by the current table exceeds a preset threshold, providing a plurality of threads for pulling the updated log data to parse the pulled log data into a message.

[0016] In a possible implementation, the method further comprises:

[0017] supplementing version information, time information and batch information of the parsed message, wherein the batch information is the order of log data update.

[0018] In a possible implementation, the method further comprises:

[0019] merging the versions of the messages in the associated multi-tables, and uploading the multi-tables after version merging to a target storage location;

[0020] Determine batch information of the uploaded log data, and mark log data corresponding to the batch information as processed for distinguishing from next batch of log data.

[0021] In a possible implementation, before the log data of the current table update is parsed into a message, the method further includes:

[0022] The log data of the current table update is cached to parse the cached log data into a message, wherein the cached content includes an index part and a data part, the index part includes an association field and a data primary key, and the data part includes the data primary key and real data.

[0023] In a possible implementation, according to the execution chain of the multi-table association, the complete message of the current table is associated to a table to be associated in the multi-table association task, including:

[0024] According to the execution chain of the multi-table association, the order of association of the current table and the table to be associated in the multi-table association task is determined.

[0025] According to the order of association with the table to be associated, it is inquired whether there is a message that can be associated with the complete message of the current table in the index part of the cached content of each table to be associated, wherein the complete message of the current table includes multiple messages of the same batch.

[0026] When there is no message that can be associated with the complete message of the current table in the index part of the cached content of the current table to be associated, it is inquired whether there is a message that can be associated with the complete message of the current table in the index part of the pre-stored source data of the current table to be associated, wherein the pre-stored source data includes an index part and a data part, the index part includes an association field and a data primary key, and the data part includes the data primary key and real data.

[0027] When there is a message that can be associated with the complete message of the current table in the index part or the pre-stored source data of the current table to be associated, the real data is inquired according to the index part or the data primary key of the index part of the current table to be associated, and the complete message of the current table is associated with the real data of the current table to be associated.

[0028] In a second aspect, embodiments of the present disclosure provide a multi-table association device, including:

[0029] A generation module is configured to parse a received multi-table association task to generate an execution chain for multi-table association.

[0030] A listening module is configured to listen to each table in the multi-table association task.

[0031] The parsing module is configured to parse the log data updated by the current table into a message when the current table is updated, and supplement information to the parsed message to obtain a complete message of the current table;

[0032] The association module is configured to associate the complete message of the current table to a table to be associated in the multi-table association task according to an execution chain of the multi-table association.

[0033] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0034] The memory is configured to store a computer program.

[0035] The processor is configured to execute the program stored on the memory to implement the multi-table association method.

[0036] In a fourth aspect, an embodiment of the present disclosure provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the multi-table association method.

[0037] The above technical solution provided by the embodiments of the present disclosure has at least some or all of the following advantages compared with the prior art:

[0038] The multi-table association method provided by the embodiments of the present disclosure parses the received multi-table association task to generate an execution chain for multi-table association, listens to each table in the multi-table association task, parses the log data updated by the current table into a message when the current table is updated, and supplements information to the parsed message to obtain a complete message of the current table, and associates the complete message of the current table to a table to be associated in the multi-table association task according to the execution chain of the multi-table association. The listening, judging, parsing and association of the complete message of the current table are performed at each node of the execution chain, and all the message processing tasks are performed in the memory without the need for middleware and the processes of serialization and deserialization, so that the message processing process occupies less disk space and consumes less disk. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings needed to be used in the embodiments or related description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0041] Figure 1 a schematic flow chart of a multi-table association method according to an embodiment of the present disclosure is shown;

[0042] Figure 2 a schematic flow chart of a multi-table association method according to another embodiment of the present disclosure is shown;

[0043] Figure 3 a schematic flow chart of a multi-table association method according to still another embodiment of the present disclosure is shown;

[0044] Figure 4 a structural schematic diagram of an execution chain for two-table association in a multi-table association method according to an embodiment of the present disclosure is shown;

[0045] Figure 5 a structural block diagram of a multi-table association apparatus according to an embodiment of the present disclosure is shown; and

[0046] Figure 6 a structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0048] Referring to Figure 1 , the embodiments of the present disclosure provide a multi-table association method, which comprises:

[0049] S1, parsing a received multi-table association task to generate an execution chain for multi-table association, wherein the execution chain comprises a plurality of nodes connected in sequence;

[0050] S2, listening to each table in the multi-table association task:

[0051] S3, parsing log data of current table update into a message, and supplementing information of the parsed message to obtain a complete message of the current table;

[0052] S4, associating the complete message of the current table to a table to be associated in the multi-table association task according to the execution chain of the multi-table association.

[0053] Referring to Figure 2 , in step S1, the execution chain for multi-table association is generated, comprising:

[0054] S21, respectively generate a corresponding execution chain for each table in the multi-table association task, to obtain an execution chain for each table;

[0055] S22, according to the association relationship between the multi-tables, connecting the execution chain of each table to generate an execution chain for multi-table association. In this embodiment, by generating an execution chain for multi-table association, when there is updated data in any one of the multi-tables, the nodes in the execution chain are executed in sequence, so that the multi-table association process is simple and fast

[0056] In this embodiment, before step S3, before the log data of the current table update is parsed into a message, the method further comprises:

[0057] When the number of log data of the current table update exceeds a preset threshold, a plurality of threads are provided for pulling the updated log data, parsing the pulled log data into a message, and quickly obtaining the updated log data.

[0058] In this embodiment, in step S3, the information of the parsed message is supplemented, including:

[0059] The version information, time information and batch information of the parsed message are supplemented to obtain a complete message, so that the complete message is associated with the multi-tables, wherein the batch information is the order of log data update.

[0060] In this embodiment, the method further comprises:

[0061] The messages in the associated multi-tables are version-merged, and the multi-tables after version-merging are uploaded to a target storage location;

[0062] The batch information of the uploaded log data is determined, and the log data corresponding to the batch information is marked as processed, to distinguish from the next batch of log data.

[0063] In this embodiment, the messages in the associated multi-tables are version-merged, which can ensure that the uploaded log data is the latest version.

[0064] In this embodiment, before step S3, before the log data of the current table update is parsed into a message, the method further comprises:

[0065] The log data of the current table update is cached to parse the cached log data into a message, wherein the cache content includes an index part and a data part, the index part includes an association field and a data primary key, and the data part includes a data primary key and real data.

[0066] In this embodiment, the updated log data is cached, which can prepare for the next association action.

[0067] Referring to Figure 3 In this embodiment, in step S4, the execution chain according to the multi-table association is used to associate the complete message of the current table with the to-be-associated tables in the multi-table association task, including:

[0068] S31, determining the order of association of the current table with the to-be-associated tables in the multi-table association task according to the execution chain of the multi-table association;

[0069] S32, querying whether there is a message that can be associated with the complete message of the current table in the index part of the cached content of each to-be-associated table according to the order of association with the to-be-associated table, wherein the complete message of the current table includes multiple messages of the same batch;

[0070] If yes, step S34 is performed;

[0071] If no, step S33 is performed;

[0072] S33, querying whether there is a message that can be associated with the complete message of the current table in the index part of the pre-stored source data of the current to-be-associated table, wherein the pre-stored source data includes an index part and a data part, the index part includes an association field and a data primary key, and the data part includes the data primary key and real data;

[0073] If yes, step S34 is performed;

[0074] If no, stop the association;

[0075] S34, querying the real data according to the index part of the current to-be-associated table or the data primary key of the index part, and associating the complete message of the current table with the real data of the current to-be-associated table.

[0076] In this embodiment, the existence of the two layers of caches of the cached content and the pre-stored source data can improve the query efficiency.

[0077] The multi-table association method of the present disclosure is applied to the process in the scene of processing a wide table of MySQL production data as follows:

[0078] First, the job information submitted by a user is parsed to generate an execution chain of each table, and each node (a MySQL instance) in the execution chain is initialized to generate an execution chain for multi-table association, wherein Figure 4 An execution chain for two-table association is provided

[0079] Second, Binlog information is obtained from a MySQL instance in real time and parsed into a Message (message)

[0080] Third, the parsed Message is supplemented with information such as site information, statistical information, and time information

[0081] Fourth, the data is sent to the corresponding node according to the execution chain for association, and the field information of the corresponding table is supplemented

[0082] Fifth, after the association is completed, the corresponding storage engine is submitted, and if the association fails, the data is discarded

[0083] Referring to Figure 4 In the execution chain for two-table association, the detailed functions of each node are as follows:

[0084] First, the initial node

[0085] The user-submitted job is parsed, and the corresponding number of nodes is generated according to the user-set parallelism, wherein the grabber, source node, merging node, and submission node can be parallel

[0086] The execution chain of each table is generated, such as the association of tables A and B

[0087] The execution chain of table A is "grabber A-source node A-merging node A-merging node B-submission node"

[0088] The execution chain of table B is "grabber B-source node B-merging node B-merging node A-submission node"

[0089] The initialization operation of each node is performed, such as the creation of a connection pool, and the execution chain for two-table association is generated

[0090] Second, the grabber node

[0091] It is mainly responsible for simulating a MySQL slave, listening to the changes of Binlog, and then parsing the Binlog to generate a Message object

[0092] In addition, if it is found that the current processing Binlog position is far from the latest Binlog position, multiple threads will be automatically generated to pull the Binlog, preventing data acquisition from becoming a bottleneck

[0093] Third, the source node

[0094] The Message object is supplemented with information such as version information and time information

[0095] The site information is maintained, that is, the wide table job is processed to which Binlog and what position, which is used for job restart or failure to continue running from the last position

[0096] Fourth, the merging node

[0097] Taking merging node A as an example

[0098] If the upstream node is the source node A, it means that A table data is changed, so the convergence node A needs to cache the data of the A table. The cache is divided into an index part and a data part. The index part k is the associated field, and v is the data primary key. The data part k is the data primary key, and v is the real data. The index part is small for each data, and more data can be cached. The data part can cache a small amount of data.

[0099] If the upstream node is the convergence node B, it means that B table data is changed, and the information of the B table data is complete. The information of A needs to be supplemented on the convergence node A. First, it is checked whether the index part is hit. If not, there is no associated data. If yes, the value of the index part is used to query the data part. If it exists, the A table data is associated and sent to the next node. If it does not exist, the primary key is used to query the source data MySQL. Due to the existence of two-layer caching, the efficiency of querying the MySQL using the primary key is not slow.

[0100] Fifth, the submission node

[0101] Responsible for submitting data to a target storage location, such as Elasticsearch, a message queue, etc.

[0102] Since the data of the A and B tables may be changed in a very short time, the version merging operation is performed on the node to prevent the old data from covering the new data.

[0103] The multi-table association method of the present disclosure is applied to a three-table association process as follows:

[0104] The data association method is expressed in sql as follows:

[0105] Select*fromaleftjoinbona.x=b.x left join c on a.y=c.y

[0106] The A table is preset as the main table, and the B and C tables are associated with the A table. The execution chains of the A, B, and C tables are different. The A table is ABC, the B table is BAC, and the C table is CAB. Only when the A table data exists can the association and storage be performed.

[0107] A detailed processing flow of A, B, and C table data updates is described

[0108] Since the B and C tables are associated with the A table, the processing flows are the same. Here, only the A and B table data updates are used to describe the multi-table association process in detail

[0109] When the A table data is updated, each node of the A table execution chain performs the following steps:

[0110] Grabber A: the binlog information of the A table change is obtained, and the Message object is generated by parsing, including operation time, operation type (update, insert, delete), data specific change, etc.

[0111] Source node A: since the wide table processing needs to record the site, the node needs to supplement the information obtained by Message, including which batch the information belongs to, and the source node will maintain the correspondence between the batch and the database site, so that when the submission node processes this batch, the information is fed back to the source node, and the source node performs site submission.

[0112] Confluence node A: receives data from source node A. Confluence node A will first cache the data in memory, and the cache is two k-v structures, one is id-index, and the other is data-index. Since a table needs to be associated with b and c tables, the association fields are different, so the id-index index has two, the first k is x field value, and the v is a id. The data-index is one, k is a id, and v is the changed data value. Then send the data to the next node confluence node B

[0113] k-v structure example:

[0114]

[0115] Confluence node B: receives data from confluence node A, and associates the corresponding data of B table according to the associated information contained. If the cache hits, supplement the B data flow in the message to confluence node C node; if the cache does not exist, query the database, if the data can be found, supplement the b data flow in the message to confluence node C node, if the b data cannot be found, directly flow the data to confluence node C node

[0116] The association between A table and B table is as follows: A table is order_table, B table is waybill_table, and order_id is used for association. After the A table data flows to confluence node B, order_id of A table is used as k of B table to query id-index v, and v is used as data-index k to query v, and then associated with A table

[0117] Converging node C: receives the data of the converging node B node, and disassociates the corresponding data of the c table according to the associated information contained, if the cache hits, the data flow of c is supplemented in the message and transferred to the submission node; if the cache does not exist, the database is queried, if the data can be found, the data flow of c is supplemented in the message and transferred to the submission node, if the data of c cannot be found, the data flow is directly transferred to the submission node.

[0118] Submission node: processes the data received by the converging node C, in the above message, the data is stored in the format of Map<String, List<>>. K is the table name, and v is the corresponding data. In the submission node, the data is merged into a Map<String, Object> k is the field name, and v is the field value. The node does not submit each data, but submits in batches to reduce network interaction. After a batch is completed, information is fed back to the source node for bit point submission.

[0119] Map<String, List<>> format is as follows:

[0120]

[0121]

[0122] Map<String, Object> format is as follows:

[0123] waybillCode: JD111111111

[0124] order_time: 2022-04-26 09:00:00

[0125] waybillFlag: 1

[0126] isCancle: 0

[0127] B table data update

[0128] Scraper B: same as above

[0129] Source node B: same as above

[0130] Converging node B: receives the data of the source node B node. The converging node B first stores the data in the memory. Since the b table only needs to be associated with the a table, the id-index index has 1, k is the x field value, and v is the id of b. The data-index is also one, k is the id of b, and v is the specific value of the changed data. Then the data is sent to the next node converging node A.

[0131] Converging node A: receives data of converging node B node, and disassociates data corresponding to a table according to contained association information, if cache hits, a data stream in the message is supplemented, and the data stream is transferred to converging node C node; if cache does not exist, data is queried in a database, if data can be found, a data stream in the message is supplemented, and the data stream is transferred to converging node C node, if a data cannot be found, the data is discarded, and when a data of a table changes, reassociation calculation is performed.

[0132] Converging node C: same as above.

[0133] Submission node: same as above.

[0134] In the multi-table association method of the disclosure, message processing of each node is performed in memory, and serialization and deserialization processes are not required. Binlog changes of a MySQL instance are limited because of the read-write capability of MySQL, so a single host can completely process the message processing, data does not need to be cached in full, only a part of data in a recent time period needs to be cached, and other data can be directly queried from a MySQL (a slave is generally configured in a production environment MySQL, and a slave Binlog is directly read and queried from the slave), which does not cause pressure on a production environment. Data is not processed in a piece-by-piece manner when being queried and stored, but in a micro-batch manner, which reduces the query pressure of the database and improves the query efficiency, and also reduces the write pressure of storage. Therefore, in a certain wide table processing scene of a warehouse report, each warehouse MySQL instance corresponds to a job, 160+ jobs are generated by using a template, 2 core cpus are used on average for each job, a total of about 300 cores are used, and disk consumption is very low. When using flink to process a wide table, data changes are collected to a message queue, and then a flink job is created for processing, 800+ cpus are used, additional cpus and disks are required for tasks of collecting the message queue, and a state also occupies a large amount of disk space. In particular, in the case of a dramatic increase in user access, the multi-table association method of the disclosure has lower delay than the flink scheme.

[0135] Referring to Figure 5 The embodiment of the disclosure provides a multi-table association device, which comprises:

[0136] A generation module 11 is configured to parse a received multi-table association task, and generate an execution chain for multi-table association;

[0137] A listening module 12 is configured to listen to each table in the multi-table association task.

[0138] a parsing module 13, configured to parse the log data updated by the current table into a message when the log data updated by the current table is parsed, and supplement information of the parsed message to obtain a complete message of the current table;

[0139] an association module 14, configured to associate the complete message of the current table to a table to be associated in a multi-table association task according to an execution chain of the multi-table association.

[0140] In this embodiment, the generating module is further configured to:

[0141] generate a corresponding execution chain for each table in the multi-table association task to obtain an execution chain of each table;

[0142] connect the execution chain of each table according to an association relationship between the multi-tables to generate an execution chain for the multi-table association.

[0143] In this embodiment, the listening module is further configured to:

[0144] when the number of the log data updated by the current table exceeds a preset threshold, provide a plurality of threads for pulling the updated log data to parse the pulled log data into a message.

[0145] In this embodiment, the parsing module is further configured to:

[0146] supplement version information, time information and batch information of the parsed message, wherein the batch information is an order of log data update.

[0147] In this embodiment, the association module is further configured to:

[0148] perform version merging on the messages in the associated multi-tables, and upload the multi-tables after version merging to a target storage location;

[0149] determine batch information of the uploaded log data, and mark the log data corresponding to the batch information as processed for distinguishing from next batch of log data.

[0150] In this embodiment, the parsing module is further configured to:

[0151] cache the log data updated by the current table to parse the cached log data into a message, wherein the cached content includes an index part and a data part, the index part includes an association field and a data primary key, and the data part includes the data primary key and real data.

[0152] In this embodiment, the association module is further configured to:

[0153] determine an order of association of the current table and a table to be associated in the multi-table association task according to an execution chain of the multi-table association.

[0154] In the index part of the cached content of each table to be associated, it is inquired whether there is a message associable with the complete message of the current table in the same batch according to the order of association with the table to be associated;

[0155] When there is no message associable with the complete message of the current table in the index part of the cached content of the current table to be associated, it is inquired whether there is a message associable with the complete message of the current table in the index part of the pre-stored source data of the current table to be associated, wherein the pre-stored source data comprises an index part and a data part, the index part comprises an association field and a data primary key, and the data part comprises the data primary key and real data;

[0156] When there is a message associable with the complete message of the current table in the index part or the pre-stored source data of the current table to be associated, the real data is inquired according to the index part or the data primary key of the index part of the current table to be associated, and the complete message of the current table is associated with the real data of the current table to be associated.

[0157] The implementation process of the functions and roles of the units in the above device is specifically described in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0158] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement it without creative labor.

[0159] In the second embodiment, any of the generation module 11, the monitoring module 12, the parsing module 13 and the association module 14 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of the other modules, and implemented in one module. At least one of the generation module 11, the monitoring module 12, the parsing module 13 and the association module 14 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or a suitable combination of any of them. Alternatively, at least one of the generation module 11, the monitoring module 12, the parsing module 13 and the association module 14 can be at least partially implemented as a computer program module that can perform the corresponding function when executed.

[0160] Referring to Figure 6 The electronic device provided by the embodiments of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 complete mutual communication through the communication bus 1140.

[0161] The memory 1130 is used to store a computer program.

[0162] The processor 1110 is used to execute the program stored in the memory 1130, and implement the multi-table association method as shown below:

[0163] The received multi-table association task is parsed to generate an execution chain for multi-table association, wherein the execution chain includes a plurality of nodes connected in sequence.

[0164] Each table in the multi-table association task is monitored.

[0165] When the current table is updated with log data, the log data updated by the current table is parsed into a message, and the parsed message is supplemented with information to obtain a complete message of the current table.

[0166] According to the execution chain of the multi-table association, the complete message of the current table is associated to the table to be associated in the multi-table association task.

[0167] The communication bus 1140 described above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0168] The communication interface 1120 is configured to communicate between the electronic device described above and other devices.

[0169] The memory 1130 can include a Random Access Memory (RAM) and can further include a non-volatile memory such as at least one disk storage. Optionally, the memory 1130 can further be at least one storage device located remotely from the aforementioned processor 1110.

[0170] The processor 1110 described above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0171] Embodiments of the present disclosure further provide a computer readable storage medium. The computer readable storage medium described above stores a computer program, and the computer program is executed by a processor to implement the multi-table association method described above.

[0172] The computer readable storage medium can be included in the device / apparatus described in the above embodiments; or it can exist separately and not be assembled into the device / apparatus. The computer readable storage medium described above carries one or more programs, and when the one or more programs are executed, the multi-table association method according to the embodiments of the present disclosure is implemented.

[0173] According to embodiments of the present disclosure, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device.

[0174] It is to be noted that the relational terms herein, such as "first" and "second", and the like, are used solely to distinguish one from another entity or action, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus.

[0175] The above descriptions are only specific embodiments of the present disclosure to enable a person skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features sought to be applied herein.

Claims

1. A multi-table join method, characterized in that, The method includes: The received multi-table join task is parsed to generate an execution chain for multi-table join, wherein the execution chain includes multiple nodes connected in sequence; Monitor each table in a multi-table join task; When the current table updates the log data, the updated log data of the current table is parsed into a message, and the parsed message is supplemented with information to obtain the complete message of the current table; According to the execution chain of multi-table joins, the complete message of the current table is joined to the table to be joined in the multi-table join task; The generation of the execution chain for multi-table joins includes: For each table in a multi-table join task, generate a corresponding execution chain to obtain the execution chain for each table; Based on the relationships between multiple tables, the execution chains of each table are connected to generate an execution chain for multi-table association.

2. The method according to claim 1, characterized in that, Before parsing the log data of the current table update into a message, the method further includes: When the amount of log data updated in the current table exceeds a preset threshold, multiple threads are provided to pull the updated log data and parse the pulled log data into messages.

3. The method according to claim 1, characterized in that, The process of supplementing the parsed message with additional information includes: The version information, time information, and batch information of the parsed message are supplemented, wherein the batch information is the order in which the log data is updated.

4. The method according to claim 3, characterized in that, The method further includes: Perform version merging on messages in the associated multi-tables, and upload the merged multi-tables to the target storage location; Determine the batch information of the uploaded log data and mark the log data corresponding to the batch information as processed to distinguish it from the next batch of log data.

5. The method according to claim 1, characterized in that, Before parsing the log data of the current table update into a message, the method further includes: The log data updated in the current table is cached so that the cached log data can be parsed into messages. The cached content includes an index part and a data part. The index part includes related fields and a primary key, and the data part includes the primary key and the actual data.

6. The method according to claim 5, characterized in that, The step of associating the complete message of the current table with the table to be associated in the multi-table association task according to the execution chain of multi-table association includes: The order in which the current table is associated with the tables to be associated in the multi-table association task is determined based on the execution chain of the multi-table association. According to the order of association with the tables to be associated, query the index part of the cached content of each table to be associated to see if there is a message that can be associated with the complete message of the current table, wherein the complete message of the current table includes multiple messages in the same batch; If there is no message in the index portion of the cached content of the current table that can be associated with the complete message of the current table, query the index portion of the pre-stored source data of the current table that can be associated with the complete message of the current table. The pre-stored source data includes an index portion and a data portion. If a message that can be associated with the complete message of the current table exists in the cached content or the index of the pre-stored data source of the current table to be associated, query the real data based on the primary key of the data in the index of the current table to be associated, and associate the complete message of the current table with the real data of the current table to be associated.

7. A multi-table association device, characterized in that, include: A generation module is used to parse the received multi-table join task and generate an execution chain for multi-table join; wherein the execution chain includes multiple nodes connected in sequence; The monitoring module is used to monitor each table in a multi-table join task. The parsing module is used to parse the updated log data of the current table into messages when the current table is updated, and to supplement the parsed messages with information to obtain the complete message of the current table. The association module is used to associate the complete message of the current table with the tables to be associated in the multi-table association task according to the execution chain of multi-table association; The generation module is also used for: For each table in a multi-table join task, generate a corresponding execution chain to obtain the execution chain for each table; Based on the relationships between multiple tables, the execution chains of each table are connected to generate an execution chain for multi-table association.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the multi-table association method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-table association method according to any one of claims 1-6.

Images