A transaction log processing method and apparatus

Abstract

The present disclosure provides a transaction log processing method and device, which relates to the technical field of data processing and can also be used in the financial field. The specific implementation scheme is as follows: traversing a pre-established lake entry operation management table according to a preset period, determining a to-be-started lake entry operation in the lake entry operation management table according to operation control information and operation state information of each lake entry operation; determining a target log cache queue corresponding to the to-be-started lake entry operation according to data source information of the to-be-started lake entry operation, and reading a preset number of target transaction logs from the target log cache queue; determining a target data lake table corresponding to the to-be-started lake entry operation according to data destination information of the to-be-started lake entry operation, and storing the target transaction logs into the target data lake table. Based on the technical scheme of the present application, the problems of difficulty in lake entry management of a large number of transaction logs and weak flow control capability can be overcome, and the processing efficiency of transaction logs can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, which can be used in the financial field, and particularly relates to a transaction log processing method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] A large amount of transaction logs will be generated during the running of an application. As a carrier of data, the transaction log is an important data source for system transaction monitoring and analysis. A common log data processing method is to use an open-source collection tool (such as flume, logstash-forwarder, etc.) to copy log data to a log cache queue, and then load it to a distributed database system for centralized management, and a distributed computing engine provides powerful big data processing capabilities.

[0003] However, in the face of the processing needs of a large amount of transaction logs, the prior art still has problems of difficult lake management and weak data flow control capability, especially when processing a large amount of backlog transaction logs, sufficient system resources are required, and if the memory resources are insufficient, the downstream work will be interrupted. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a transaction log processing method and device, which can overcome the problems of difficult lake management and weak flow control capability of a large amount of transaction logs, and effectively improve the processing efficiency of transaction logs.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a transaction log processing method, which comprises:

[0007] traversing a pre-established lake job management table according to a preset period, determining a to-be-started lake job in the lake job management table according to job control information and job state information of each lake job;

[0008] determining a target log cache queue corresponding to the to-be-started lake job according to data source information of the to-be-started lake job, and reading a preset number of target transaction logs from the target log cache queue;

[0009] determining a target data lake table corresponding to the to-be-started lake job according to data destination information of the to-be-started lake job, and storing the target transaction logs to the target data lake table.

[0010] In some optional modes of the embodiment, the step of establishing the lake job management table comprises:

[0011] obtaining an initial transaction log and determining an application attribute of the initial transaction log;

[0012] adding the initial transaction log to a corresponding initial log cache queue according to the application attribute, wherein a topic attribute of the initial log cache queue is mapped to the application attribute of the initial transaction log one by one;

[0013] configuring an in-lake job in an in-lake job management table according to the initial log cache queue, wherein the in-lake job includes data source information, data destination information, job control information, job state information and job heartbeat information;

[0014] and the method further comprises: in the data lake, an initial data lake table corresponding to the topic attribute of the initial log cache queue is established in advance.

[0015] In some optional manners of the embodiment, in response to the data source information of the to-be-started in-lake job including a target address attribute of the target log cache queue, the determining of the target log cache queue corresponding to the to-be-started in-lake job according to the data source information of the to-be-started in-lake job comprises:

[0016] determining an initial log cache queue matched with the target address attribute from the initial log cache queue, and taking the matched initial log cache queue as the target log cache queue;

[0017] In response to the data destination information of the to-be-started in-lake job including a to-be-started data lake table name attribute, the determining of the target data lake table corresponding to the to-be-started in-lake job according to the data destination information of the to-be-started in-lake job comprises:

[0018] determining an initial data lake table matched with the to-be-started data lake table name attribute from the initial data lake table, and taking the matched initial data lake table as the target data lake table.

[0019] In some optional manners of the embodiment, after the determining of the to-be-started in-lake job in the in-lake job management table, the method further comprises:

[0020] the in-lake management node distributes the to-be-started in-lake job to a corresponding in-lake working node;

[0021] and when the in-lake job is configured in the in-lake job management table, an initial value of the job control information is configured as “1” and an initial value of the initial job state information is configured as “0”;

[0022] and after the in-lake working node receives the to-be-started in-lake job, the job state information of the to-be-started in-lake job is switched from “0” to “1”.

[0023] In some optional embodiments of this example, the method further includes:

[0024] The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operation in operation in the lake entry operation management table is determined.

[0025] In response to determining that the time difference between the previous heartbeat information and the current heartbeat information of the in-process lake entry operation exceeds a preset time threshold, it is determined that there is an anomaly in the in-process lake entry operation;

[0026] The lake entry management node switches the operation control information of the in-operation lake entry operation from "1" to "0" and sends a stop operation command to the lake entry work node corresponding to the in-operation lake entry operation.

[0027] In some optional embodiments of this example, the method further includes:

[0028] The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operations to be stopped in the lake entry operation management table are determined.

[0029] The lake entry management node sends a stop operation command to the lake entry work node corresponding to the lake entry operation to be stopped.

[0030] In some optional embodiments of this example, after sending the stop operation command, the method further includes:

[0031] In response to the lake entry work node being in a working state, after the lake entry work node completes its current work task, the work status information of the abnormal lake entry operation or the lake entry operation to be stopped will be switched from "1" to "0".

[0032] In response to the lake entry work node being in an idle state, the work status information of the abnormal lake entry operation or the lake entry operation to be stopped is directly switched from "1" to "0".

[0033] In some optional embodiments of this example, the steps of determining the lake entry operation to be started, the lake entry operation in progress, and the lake entry operation to be stopped include:

[0034] In response to the operation control information of the lake entry operation being "1" and the operation status information being "0", the lake entry operation is determined to be a lake entry operation to be started;

[0035] In response to the operation control information of the lake entry operation being "1" and the operation status information being "1", the lake entry operation is determined to be an in-process lake entry operation;

[0036] In response to the operation control information of the lake entry operation being "0", the lake entry operation is determined to be a lake entry operation to be stopped.

[0037] Secondly, this application provides a transaction log processing apparatus, the apparatus comprising:

[0038] The module for determining lake entry operations to be started is configured to traverse a pre-established lake entry operation management table according to a preset cycle, and determine the lake entry operations to be started in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation.

[0039] The target transaction log reading module is configured to determine the target log cache queue corresponding to the lake entry operation to be started based on the data source information of the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue.

[0040] The target transaction log storage module is configured to determine the target data lake table corresponding to the lake entry operation to be started based on the data destination information of the lake entry operation to be started, and store the target transaction log in the target data lake table.

[0041] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the transaction log processing method.

[0042] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the transaction log processing method.

[0043] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the transaction log processing method.

[0044] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0045] To address the problems in the existing technology, the transaction log processing method and apparatus provided in this application can continuously identify the pending inbound operations by traversing the inbound operation management table according to a preset period, and store a preset number of transaction logs into the data lake based on the data source information and data destination information of the pending inbound operations. This overcomes the problems of difficulty in managing a large number of transaction logs in the data lake and weak flow control capabilities, and realizes continuous synchronization of transaction logs and data lake updates, effectively improving the processing efficiency of transaction logs. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0047] Figure 1 This is an exemplary system architecture diagram in which one embodiment of this application can be applied;

[0048] Figure 2 This is a flowchart of one embodiment of the transaction log processing method according to this application;

[0049] Figure 3 This is a flowchart of a method for establishing a lake entry operation management table according to an embodiment of this application;

[0050] Figure 4 This is a flowchart of a method for handling lake entry operations in operation, according to an embodiment of this application;

[0051] Figure 5 This is a flowchart of a method for handling the cessation of lake entry operations according to an embodiment of this application;

[0052] Figure 6 This is a flowchart of a processing method based on a stop operation command according to an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of one embodiment of the transaction log processing apparatus according to this application;

[0054] Figure 8 This is a block diagram of an electronic device used to implement the transaction log processing method of the embodiments of this application. Detailed Implementation

[0055] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0056] Figure 1 An exemplary system architecture is shown, illustrating an embodiment of the transaction log processing method to which the present disclosure can be applied. For example... Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0057] Terminal 101 can interact with server 103 via network 102. Terminal 101 stores transaction logs to be processed. Server 103 can obtain the transaction logs to be processed from terminal 101, add the transaction logs to the corresponding log cache queue, and finally store the transaction logs in the data lake.

[0058] It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0059] It should be understood that Figure 1 The number of terminals, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminals, networks, and servers can be included.

[0060] It should be noted that the transaction log processing method provided in this embodiment is generally executed by server 103, and correspondingly, the transaction log processing device is generally set in server 103.

[0061] Figure 2 A schematic flowchart illustrating a transaction log processing method according to an embodiment of this disclosure is shown, such as... Figure 2 As shown, the method includes the following steps:

[0062] Step 201: Traverse the pre-established lake entry operation management table according to the preset cycle, and determine the lake entry operations to be started in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation;

[0063] Step 202: Based on the data source information of the lake entry operation to be started, determine the target log cache queue corresponding to the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue;

[0064] Step 203: Based on the data destination information of the lake entry operation to be started, determine the target data lake table corresponding to the lake entry operation to be started, and store the target transaction log in the target data lake table.

[0065] This embodiment continuously identifies pending inbound operations by traversing the inbound operation management table according to a preset cycle. Based on the data source and destination information of the pending inbound operations, a preset number of transaction logs are stored in the data lake. This overcomes the existing problems of managing a large number of transaction logs in the data lake and weak flow control capabilities, and achieves continuous synchronization of transaction logs and data lake updates, effectively improving the processing efficiency of transaction logs.

[0066] The following is about Figure 2 Each step in the process will be explained in detail.

[0067] Step 201: Traverse the pre-established lake entry operation management table according to the preset cycle, and determine the lake entry operations to be started in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation.

[0068] In this embodiment, the lake entry operation management form will be introduced first:

[0069] The lake entry operation management table stores multiple lake entry operations. Each lake entry operation includes data source information and data destination information. The data source information is the service address and topic of the log cache queue to be read, and the data destination information is the storage destination, such as the data lake database name and data table name. These lake entry operations are recorded in one table, which is the lake entry operation management table.

[0070] In some optional embodiments of this example, such as Figure 3 As shown, the steps for pre-establishing the lake entry operation management table include:

[0071] Step 301: Obtain the initial transaction log and determine the application attributes of the initial transaction log.

[0072] In this embodiment, transaction logs are acquired through two methods: intrusive collection and non-intrusive collection. Intrusive collection embeds the transaction log code logic into the application code. Its advantages include avoiding the log files from being written to disk, reducing storage costs and end-to-end latency, and improving the timeliness of subsequent log monitoring. The disadvantage is that it requires modification of the application's business logic. Non-intrusive collection monitors updates to the initial transaction log file and promptly replicates the updated transaction log data to a distributed log cache queue.

[0073] It should be noted that in practical applications, if timeliness is a critical requirement, an intrusive data collection method should be chosen; if code modification is not permitted, a non-intrusive data collection method should be chosen. This application does not impose any restrictions on this.

[0074] Furthermore, after obtaining the transaction logs, it is also necessary to determine the application attributes of the transaction logs. In order to distinguish the transaction logs obtained initially from those to be stored later, this embodiment names the transaction logs obtained initially as the initial transaction logs and the transaction logs to be stored later as the target transaction logs.

[0075] The application attribute of the transaction log can be understood as the application that generated the transaction log. For example, the application attribute can include personal online banking application, credit card application, and personal account application.

[0076] Step 302: According to the application attributes, add the initial transaction log to the corresponding initial log cache queue, wherein the topic attributes of the initial log cache queue are mapped one-to-one with the application attributes of the initial transaction log.

[0077] In this embodiment, the log caching queue is mainly responsible for caching transaction log data to be added to the data lake. Transaction logs are stored in different message topic log caching queues according to different application attributes. The distributed log caching queue of this application adopts Kafka technology, which integrates log storage and streaming data processing, and is exposed to the downstream real-time data lake through a unified interface service. It also features message persistence, high throughput, and high timeliness.

[0078] In other words, each log cache queue has a different theme. Using the previous example, the theme attributes of the log cache queue can include personal online banking, credit card, and personal account themes. To distinguish it from subsequent log cache queues, the log cache queue used during the lake entry operation management table creation phase is named the initial log cache queue, and the log cache queue used during subsequent lake entry tasks is named the target log cache queue.

[0079] In this embodiment, the topic attributes of the initial log cache queue are mapped one-to-one with the attributes of the initial transaction logs. For example, the personal online banking topic is mapped to the personal online banking application, the credit card topic to the credit card application, and the personal account topic to the personal account application. Therefore, based on this mapping relationship, the initial transaction logs can be added to the corresponding initial log cache queue according to the application attributes. For example, the initial transaction logs of the personal online banking application can be added to the initial log cache queue with the topic attribute of "personal online banking".

[0080] It should be noted that after determining the log attributes of the initial log cache queue, the method in this example also includes the following steps not shown in the figure: in the data lake, an initial data lake table corresponding to the topic attributes of the initial log cache queue is pre-established.

[0081] Using the previous example, for different attributes of the initial log cache queue, corresponding initial data lake tables are created in the data lake. For example, an initial data lake table is created corresponding to the initial log cache queue of the personal online banking theme, and its table name is "Personal Online Banking Theme Data Lake Table"; an initial data lake table is created corresponding to the initial log cache queue of the credit card theme, and its table name is "Credit Card Theme Data Lake Table"; and an initial data lake table is created corresponding to the initial log cache queue of the personal account theme, and its table name is "Personal Account Theme Data Lake Table".

[0082] Step 303: Configure the lake entry operation in the lake entry operation management table according to the initial log cache queue. The lake entry operation includes data source information, data destination information, operation control information, operation status information and operation heartbeat information.

[0083] In this embodiment, the lake entry operation is further configured in the lake entry operation management table according to the initial log cache queue. In addition to the data source information and data destination information in the previous example, the lake entry operation also includes operation control information, lake entry operation status information and lake entry operation heartbeat information.

[0084] Before introducing job control information, inbound job status information, and inbound job heartbeat information, we introduce inbound management nodes and inbound worker nodes. Inbound worker nodes execute inbound jobs, specifically responsible for reading transaction logs from the log cache queue and storing them in the data lake. Inbound management nodes manage multiple inbound worker nodes and multiple inbound jobs to improve the efficiency of transaction log processing by the inbound worker nodes and overcome the shortcomings of existing inbound management technologies. The job control information value can be "0" or "1". When it is "0", it indicates that the inbound management node sends a start job command to the inbound worker node; when it is "1", it indicates that the inbound management node sends a stop job command to the inbound worker node. The job status information value can also be "0" or "1". When it is "0", it indicates that the inbound job has not been run; when it is "1", it indicates that the inbound job is running. Inbound worker nodes periodically send job heartbeat information to the inbound management node during their operation.

[0085] It should be noted that when configuring a new lake entry operation in the lake entry operation management table, the initial value of its operation control information is configured as "1", and the initial value of the initial operation status information is configured as "0".

[0086] At this point, the lake entry operation management form has been completed.

[0087] Based on this, this embodiment traverses the pre-established lake entry operation management table according to a preset cycle, and determines the lake entry operations to be started in the lake entry operation management table according to the operation control information and operation status information of each lake entry operation.

[0088] It should be noted that this application does not limit the preset period, which can be customized by those skilled in the art according to the current memory usage. For example, when memory resources are scarce, the preset period can be increased to ensure that there are relatively sufficient system resources to process transaction logs; when memory resources are abundant, the preset period can be decreased to improve the efficiency of transaction log processing and effectively enhance the flow control capability of transaction logs.

[0089] It should be understood that the three operations of acquiring transaction logs, caching the log cache queue, and storing transaction logs in the data lake are performed synchronously. Therefore, the inbound job management table should include inbound jobs that have not been processed, inbound jobs that are being processed, and inbound jobs that need to be stopped. Among them, inbound jobs that have not been processed are those waiting to be started; inbound jobs that are being processed are those running; and inbound jobs that need to be stopped are those waiting to be stopped. The job control information and job status information of these three different types of inbound jobs are different.

[0090] In some optional embodiments of this example, the steps of determining the lake entry operation to be started, the lake entry operation in progress, and the lake entry operation to be stopped include:

[0091] In response to the operation control information of the lake entry operation being "1" and the operation status information being "0", the lake entry operation is determined to be a lake entry operation to be started; in response to the operation control information of the lake entry operation being "1" and the operation status information being "1", the lake entry operation is determined to be a lake entry operation in operation; in response to the operation control information of the lake entry operation being "0", the lake entry operation is determined to be a lake entry operation to be stopped.

[0092] In this embodiment, the lake entry management node periodically scans the lake entry operation table to find lake entry operations to be processed, starts the lake entry operation thread, and at the same time finds lake entry operations that have stopped working and stops the corresponding lake entry operation thread.

[0093] Step 202: Based on the data source information of the lake entry operation to be started, determine the target log cache queue corresponding to the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue.

[0094] It should be understood that, regardless of whether it is a lake entry operation to be started, a lake entry operation in operation, or a lake entry operation to be stopped, it includes the data source information, data destination information, operation status information, operation control information, and operation heartbeat information of the aforementioned examples.

[0095] In some optional embodiments of this example, in response to the data source information of the lake entry operation to be initiated including the target address attribute of the target log cache queue, step 202 further includes:

[0096] From the initial log cache queue, determine the initial log cache queue that matches the target address attribute, and use the matching initial log cache queue as the target log cache queue.

[0097] In other words, from the aforementioned initial log cache queue, the initial log cache queue whose address attribute matches the target address attribute is determined, and the matching initial log cache queue is determined as the target log cache queue. It should be understood that the target log cache queue stores transaction logs that are to be stored in the data lake.

[0098] Therefore, a preset number of target transaction logs are read from the target log cache queue. It should be noted that this application does not limit the preset number, that is, the number of data entries read from the target log cache queue each time can be configured before startup. It should be understood that the number of data entries determines the speed of data entry into the lake. Those skilled in the art can determine an appropriate number of data entries based on the current actual situation, such as the system resource memory usage, to ensure that there are relatively sufficient system resources to process transaction logs, thereby effectively improving the flow control capability of transaction logs.

[0099] Step 203: Based on the data destination information of the lake entry operation to be started, determine the target data lake table corresponding to the lake entry operation to be started, and store the target transaction log in the target data lake table.

[0100] In some optional embodiments of this example, in response to the data destination information of the lake entry operation to be initiated including the lake surface name attribute, step 203 further includes:

[0101] From the initial data lake table, determine an initial data lake table that matches the name attribute of the data lake table to be started, and use the matching initial data lake table as the target data lake table.

[0102] Using the example above as an example, if the name attribute of the data lake table to be started is "Personal Online Banking Theme Data Lake Table", then the initial data lake table with the same name as "Personal Online Banking Theme Data Lake Table" will be used as the target data lake table, and the preset number of target transaction logs read above will be stored in the target data lake table to complete one entry operation.

[0103] It should be noted that during data synchronization into the data lake, a specified number of data entries are read each time, and the processing of the next batch (next preset number of data entries) is only performed after the current batch (current preset number of data entries) has been successfully written into the data lake.

[0104] It should also be noted that the data lake in this application adopts the Hudi open-source data lake architecture. By introducing the Spark computing framework and connecting with the upstream log cache queue, the data of the log cache queue is centrally stored in the real-time data lake. The difference between the real-time data lake and the log cache queue is: the log cache queue only retains log data within the recent specified number of days, and data older than that date is automatically cleaned up; the real-time data lake can store long-term log data and supports dynamic updates of transaction log data, and can accept high-concurrency queries.

[0105] In some optional embodiments of this example, after determining the lake entry operation to be started in the lake entry operation management table, the transaction log processing method further includes: the lake entry management node distributing the lake entry operation to be started to the corresponding lake entry work node.

[0106] As mentioned above, the operation control information for a pending lake entry operation is "1", and the operation status information is "0". After the lake entry management node finds these pending lake entry operations, it packages them into tasks and distributes them to the lake entry work nodes. It should be understood that the number of pending lake entry operations is not unique, and there is a one-to-one correspondence between the lake entry work nodes and the lake entry operations. Subsequently, the lake entry work nodes are responsible for the specific lake entry work.

[0107] It should be noted that after the lake entry work node receives the lake entry operation to be started, it switches the operation status information of the lake entry operation to be started from "0" to "1".

[0108] To further improve the management of transaction logs entering the lake, in some optional methods of this embodiment, such as... Figure 4 As shown, the methods for processing transaction logs also include:

[0109] Step 401: Traverse the pre-established lake entry operation management table according to the preset cycle, and determine the lake entry operations in operation in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation;

[0110] Step 402: In response to determining that the time difference between the previous operation heartbeat information and the current operation heartbeat information of the in-process lake entry operation exceeds a preset time threshold, it is determined that there is an anomaly in the in-process lake entry operation;

[0111] Step 403: The lake entry management node switches the operation control information of the in-run lake entry operation from "1" to "0" and sends a stop operation command to the lake entry work node corresponding to the in-run lake entry operation.

[0112] Based on the aforementioned method for determining in-process lake entry operations, this application will not elaborate further. During operation, each lake entry work node periodically sends heartbeat information of its in-process operations to the lake entry management node. When the lake entry management node periodically scans (traverses according to a preset cycle) the lake entry operation management table, if it finds that the time difference between the previous heartbeat information and the current heartbeat information of an in-process lake entry operation exceeds a preset time threshold (for example, if the most recent heartbeat time and the current time exceed a certain time difference), it considers the in-process lake entry operation to be abnormal. The lake entry management node needs to switch the operation control information of the abnormal in-process lake entry operation from "1" to "0", thereby terminating the in-process lake entry operation and sending a stop operation command to the lake entry work node corresponding to the in-process lake entry operation.

[0113] This embodiment can locate abnormal inbound operations and terminate them in a timely manner, releasing memory resources, avoiding resource waste caused by abnormal inbound operations, and further improving the processing efficiency of transaction logs.

[0114] In the foregoing Figure 4 Based on the aforementioned method, to further improve the management of transaction logs entering the lake, some optional methods in this embodiment, such as... Figure 5 As shown, the methods for processing transaction logs also include:

[0115] Step 501: Traverse the pre-established lake entry operation management table according to the preset cycle, and determine the lake entry operations to be stopped in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation.

[0116] In this embodiment, the method for determining the lake entry operation to be stopped can refer to the aforementioned example, and will not be described in detail here.

[0117] Step 502: The lake entry management node sends a stop operation command to the lake entry work node corresponding to the lake entry operation to be stopped.

[0118] In the foregoing Figure 4 Based on the method, it can be seen that the jobs to be stopped are actually the in-run jobs that are abnormal. In order to improve the accuracy of transaction log processing, it is necessary to traverse the pre-established in-run job management table according to the preset cycle, identify the in-run jobs to be stopped, and send a stop job command to the in-run work node corresponding to the job to be stopped.

[0119] In some optional embodiments of this example, such as Figure 6 As shown, after sending the stop job command, the transaction log processing method also includes steps 601 and 602. For jobs that need to be stopped from entering the lake, steps 601-602 aim to "gracefully" take the lake entry jobs of the lake entry management node offline. That is, the stop job command does not occur instantaneously. Specifically:

[0120] Step 601: In response to the lake entry work node being in a working state, after the lake entry work node completes its current work task, switch the work status information of the abnormal lake entry operation or the lake entry operation to be stopped from "1" to "0".

[0121] In this embodiment, when the data entry node receives a stop operation instruction, if it is in the process of entering the data into the lake for the current batch, that is, the target transaction log data of the current batch has not yet been fully stored in the target data lake, then after the data is entered into the lake, the operation status information of the corresponding operation in the data entry operation management table will be set to "0". For example, the operation status information of the abnormal running data entry operation or the data entry operation to be stopped will be switched from "1" to "0".

[0122] Step 602: In response to the lake entry work node being in an idle state, directly switch the work status information of the abnormal lake entry operation or the lake entry operation to be stopped from "1" to "0".

[0123] In this embodiment, when the entry node receives a stop operation instruction, if it is in an idle state, that is, the target transaction log data of the current batch of the preset quantity has been stored in the target data lake, the operation status information of the corresponding operation in the entry operation management table can be directly set to "0". For example, the operation status information of the abnormal running entry operation or the entry operation to be stopped can be switched from "1" to "0".

[0124] In this embodiment, the timing for modifying the job status information can be determined based on the status of the lake entry work node, thereby enabling the lake entry work node to be "gracefully" taken offline at the lake entry management node, effectively improving the stability of the entire transaction log processing method.

[0125] This application aggregates information from all data entry operations into the lake, overcoming existing problems such as difficulty in managing multi-source data entering the lake and weak flow control capabilities. It provides a real-time scheduling solution for data entering the lake, improving data processing efficiency and strengthening monitoring capabilities.

[0126] Based on the same inventive concept, this application also provides a transaction log processing apparatus, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle by which this transaction log processing apparatus solves the problem is similar to that of the transaction log processing method, the implementation of the transaction log processing apparatus can refer to the implementation of the transaction log processing method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0127] like Figure 7 As shown, the transaction log processing device includes:

[0128] The lake entry operation determination module 701 is configured to traverse a pre-established lake entry operation management table according to a preset cycle, and determine the lake entry operations to be started in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation.

[0129] The target transaction log reading module 702 is configured to determine the target log cache queue corresponding to the lake entry operation to be started based on the data source information of the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue.

[0130] The target transaction log storage module 703 is configured to determine the target data lake table corresponding to the lake entry operation to be started based on the data destination information of the lake entry operation to be started, and store the target transaction log in the target data lake table.

[0131] In some optional embodiments of this example, the step of establishing the lake entry operation management table includes:

[0132] Obtain the initial transaction log and determine the application attributes of the initial transaction log;

[0133] Based on the application attributes, the initial transaction log is added to the corresponding initial log cache queue, wherein the topic attributes of the initial log cache queue are mapped one-to-one with the application attributes of the initial transaction log;

[0134] Based on the initial log cache queue, configure the lake entry operation in the lake entry operation management table, wherein the lake entry operation includes data source information, data destination information, operation control information, operation status information, and operation heartbeat information;

[0135] The device is further configured to: pre-create an initial data lake table in the data lake corresponding to the topic attributes of the initial log cache queue.

[0136] In some optional embodiments of this example, in response to the data source information of the lake entry operation to be started including the target address attribute of the target log cache queue, the step of determining the target log cache queue corresponding to the lake entry operation to be started based on the data source information of the lake entry operation to be started includes:

[0137] From the initial log cache queue, determine the initial log cache queue that matches the target address attribute, and use the matching initial log cache queue as the target log cache queue;

[0138] In response to the data destination information of the lake entry operation to be initiated, which includes the name attribute of the lake table to be entered, the step of determining the target lake table corresponding to the lake entry operation to be initiated based on the data destination information of the lake entry operation to be initiated includes:

[0139] From the initial data lake table, determine an initial data lake table that matches the name attribute of the data lake table to be started, and use the matching initial data lake table as the target data lake table.

[0140] In some alternative embodiments of this embodiment, after determining the lake entry operation to be initiated in the lake entry operation management table, the device is further configured to:

[0141] The lake entry management node distributes the lake entry operation to be started to the corresponding lake entry work node;

[0142] When configuring lake entry operations in the lake entry operation management table, the initial value of the operation control information is configured as "1", and the initial value of the initial operation status information is configured as "0".

[0143] And after receiving the lake entry operation to be started at the lake entry work node, the operation status information of the lake entry operation to be started is switched from "0" to "1".

[0144] In some alternative embodiments of this embodiment, the device is further configured as follows:

[0145] The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operation in operation in the lake entry operation management table is determined.

[0146] In response to determining that the time difference between the previous heartbeat information and the current heartbeat information of the in-process lake entry operation exceeds a preset time threshold, it is determined that there is an anomaly in the in-process lake entry operation;

[0147] The lake entry management node switches the operation control information of the in-operation lake entry operation from "1" to "0" and sends a stop operation command to the lake entry work node corresponding to the in-operation lake entry operation.

[0148] In some alternative embodiments of this embodiment, the device is further configured as follows:

[0149] The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operations to be stopped in the lake entry operation management table are determined.

[0150] The lake entry management node sends a stop operation command to the lake entry work node corresponding to the lake entry operation to be stopped.

[0151] In some alternative embodiments of this example, after sending a stop operation command, the device is further configured to:

[0152] In response to the lake entry work node being in a working state, after the lake entry work node completes its current work task, the work status information of the abnormal lake entry operation or the lake entry operation to be stopped will be switched from "1" to "0".

[0153] In response to the lake entry work node being in an idle state, the work status information of the abnormal lake entry operation or the lake entry operation to be stopped is directly switched from "1" to "0".

[0154] In some optional embodiments of this example, the steps of determining the lake entry operation to be started, the lake entry operation in progress, and the lake entry operation to be stopped include:

[0155] In response to the operation control information of the lake entry operation being "1" and the operation status information being "0", the lake entry operation is determined to be a lake entry operation to be started;

[0156] In response to the operation control information of the lake entry operation being "1" and the operation status information being "1", the lake entry operation is determined to be an in-process lake entry operation;

[0157] In response to the operation control information of the lake entry operation being "0", the lake entry operation is determined to be a lake entry operation to be stopped.

[0158] It should be noted that the acquisition, storage, use and processing of data involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0159] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the transaction log processing method of the foregoing embodiments.

[0161] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the transaction log processing method of the foregoing embodiments.

[0162] A computer program product includes a computer program that, when executed by a processor, implements the transaction log processing method of the aforementioned embodiments.

[0163] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] likeFigure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0165] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the transaction log processing method. For example, in some embodiments, the transaction log processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the transaction log processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the transaction log processing method by any other suitable means (e.g., by means of firmware).

[0167] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0172] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0173] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing transaction logs, characterized in that, include: The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operation to be started in the lake entry operation management table is determined. Based on the data source information of the lake entry operation to be started, determine the target log cache queue corresponding to the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue; Based on the data destination information of the lake entry operation to be started, determine the target data lake table corresponding to the lake entry operation to be started, and store the target transaction log in the target data lake table; The steps for establishing the lake entry operation management table include: Obtain the initial transaction log and determine the application attributes of the initial transaction log; Based on the application attributes, the initial transaction log is added to the corresponding initial log cache queue, wherein the topic attributes of the initial log cache queue are mapped one-to-one with the application attributes of the initial transaction log; Based on the initial log cache queue, configure the lake entry operation in the lake entry operation management table, wherein the lake entry operation includes data source information, data destination information, operation control information, operation status information, and operation heartbeat information; The method further includes: in the data lake, pre-establishing an initial data lake table corresponding to the topic attributes of the initial log cache queue; After determining the lake entry operations to be initiated in the lake entry operation management table, the method further includes: The lake entry management node distributes the lake entry operation to be started to the corresponding lake entry work node; When configuring lake entry operations in the lake entry operation management table, the initial value of the operation control information is configured as "1", and the initial value of the operation status information is configured as "0". And after the lake entry work node receives the lake entry operation to be started, it switches the operation status information of the lake entry operation to be started from "0" to "1".

2. The method according to claim 1, characterized in that, In response to the data source information of the lake entry operation to be initiated including the target address attribute of the target log cache queue, the step of determining the target log cache queue corresponding to the lake entry operation to be initiated based on the data source information of the lake entry operation to be initiated includes: From the initial log cache queue, determine the initial log cache queue that matches the target address attribute, and use the matching initial log cache queue as the target log cache queue; In response to the data destination information of the lake entry operation to be initiated, which includes the name attribute of the lake table to be entered, the step of determining the target lake table corresponding to the lake entry operation to be initiated based on the data destination information of the lake entry operation to be initiated includes: From the initial data lake table, determine an initial data lake table that matches the name attribute of the data lake table to be started, and use the matching initial data lake table as the target data lake table.

3. The method according to claim 1, characterized in that, The method further includes: The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operation in operation in the lake entry operation management table is determined. In response to determining that the time difference between the previous heartbeat information and the current heartbeat information of the in-process lake entry operation exceeds a preset time threshold, it is determined that there is an anomaly in the in-process lake entry operation; The lake entry management node switches the operation control information of the in-run lake entry operation from "1" to "0" and sends a stop operation command to the lake entry work node corresponding to the in-run lake entry operation.

4. The method according to claim 3, characterized in that, The method further includes: The pre-established lake entry operation management table is traversed according to a preset cycle. Based on the operation control information and operation status information of each lake entry operation, the lake entry operations to be stopped in the lake entry operation management table are determined. The lake entry management node sends a stop operation command to the lake entry work node corresponding to the lake entry operation to be stopped.

5. The method according to claim 3 or 4, characterized in that, After sending the stop job command, the method further includes: In response to the lake entry work node being in a working state, after the lake entry work node completes its current work task, the work status information of abnormal lake entry operations or lake entry operations to be stopped will be switched from "1" to "0". In response to the lake entry work node being in an idle state, the work status information of the abnormal lake entry operation or the lake entry operation to be stopped is directly switched from "1" to "0".

6. The method according to claim 5, characterized in that, The steps for determining the lake entry operation to be started, the lake entry operation in operation, and the lake entry operation to be stopped include: In response to the operation control information of the lake entry operation being "1" and the operation status information being "0", the lake entry operation is determined to be a lake entry operation to be started; In response to the operation control information of the lake entry operation being "1" and the operation status information being "1", the lake entry operation is determined to be an in-process lake entry operation; In response to the operation control information of the lake entry operation being "0", the lake entry operation is determined to be a lake entry operation to be stopped.

7. A transaction log processing apparatus, characterized in that, include: The module for determining lake entry operations to be started is configured to traverse a pre-established lake entry operation management table according to a preset cycle, and determine the lake entry operations to be started in the lake entry operation management table based on the operation control information and operation status information of each lake entry operation. The target transaction log reading module is configured to determine the target log cache queue corresponding to the lake entry operation to be started based on the data source information of the lake entry operation to be started, and read a preset number of target transaction logs from the target log cache queue. The target transaction log storage module is configured to determine the target data lake table corresponding to the lake entry operation to be started based on the data destination information of the lake entry operation to be started, and store the target transaction log in the target data lake table; The steps for establishing the lake entry operation management table include: Obtain the initial transaction log and determine the application attributes of the initial transaction log; Based on the application attributes, the initial transaction log is added to the corresponding initial log cache queue, wherein the topic attributes of the initial log cache queue are mapped one-to-one with the application attributes of the initial transaction log; Based on the initial log cache queue, configure the lake entry operation in the lake entry operation management table, wherein the lake entry operation includes data source information, data destination information, operation control information, operation status information, and operation heartbeat information; The device is further configured to: pre-create an initial data lake table in the data lake corresponding to the topic attributes of the initial log cache queue; After determining the lake entry operations to be initiated in the lake entry operation management table, the device is further configured as follows: The lake entry management node distributes the lake entry operation to be started to the corresponding lake entry work node; When configuring lake entry operations in the lake entry operation management table, the initial value of the operation control information is configured as "1", and the initial value of the operation status information is configured as "0". And after the lake entry work node receives the lake entry operation to be started, it switches the operation status information of the lake entry operation to be started from "0" to "1".

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the transaction log processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of a transaction log processing method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the transaction log processing method according to any one of claims 1 to 6.

Images