Data change log collection method and device under distributed system, medium and equipment

Abstract

The embodiment of the invention discloses a data change log collection method and device under a distributed system, a medium and equipment, and relates to the technical field of data change recording, and the method comprises the steps: in response to a change recording starting instruction initiated by a user, executing an operation conforming to a change recording rule on a working node, and obtaining execution data; before the transaction is submitted, execution data are sent to a target node, and message registration is carried out; wherein the target node is any node in the distributed system; and when the transaction is submitted, in response to a transaction submission notification initiated by the working node and received by the target node, carrying out a change log collection task at the target node. According to the method and the device, the change record task is accepted through any node in the distributed system, so that resources occupied by the main node in the distributed system for maintaining the change record message are liberated, and change log data disk dropping is not required to be completed before the transaction is submitted; the target node only starts to change the recorded task after receiving the transaction submission notification, so that redundant processing caused by transaction rollback is reduced, and the problem of transaction time delay caused by waiting for change log dropping before transaction submission is solved.

Description

Technical Field

[0001] The present application relates to the technical field of data change recording, and in particular to a method, device, medium and equipment for collecting data change logs in a distributed system. Background Art

[0002] Data change recording is a method commonly used in the field of database technology for incremental data backup processing. The general technical implementation route is mainly based on WAL logs. Given the scarcity of network resources in distributed database systems, the communication overhead between nodes is one of the main technical points to consider in the design of distributed systems today. Recording changes to each node within the cluster based on WAL logs will inevitably introduce the cost of synchronization and maintenance of WAL logs between master and slave nodes, increasing the communication and maintenance overhead of the master node. In addition, WAL logs require that data must be written to disk before a transaction is committed. Although WAL logs use sequential IO plus page cache to maximize write performance, they still cannot solve the impact of data change records on transaction execution latency. Summary of the invention

[0003] The main purpose of the present application is to provide a method, device, medium and equipment for collecting data change logs in a distributed system, aiming to solve the problem of the impact of data change records in a distributed system on transaction execution delay in the prior art.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for collecting data change logs in a distributed system, comprising the following steps:

[0006] In response to a change record start instruction initiated by a user, an operation that complies with the change record rule is executed on a working node and execution data is obtained;

[0007] Before the transaction is committed, the execution data is sent to the target node and the message is registered; wherein the target node is any node in the distributed system;

[0008] When a transaction is committed, in response to the target node receiving a transaction commit notification initiated by the working node, a change log collection task is performed at the target node.

[0009] In a possible implementation manner of the first aspect, sending the execution data to the target node and performing message registration includes:

[0010] Send the execution data to the target node, and perform hash calculation on the transaction number to which the change message belongs based on the change message of the execution data to obtain the hash key value;

[0011] According to the hash key value, obtain the change record and write it into the partition;

[0012] Write the partition according to the change record and register the information.

[0013] In a possible implementation of the first aspect, when a transaction is committed, in response to the target node receiving a transaction commit notification initiated by the working node, performing a change log collection task at the target node includes:

[0014] When a transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, the target transaction number and the target message source node are obtained according to the hash key value;

[0015] The registered change message with the same target transaction number and target message source node is sent to the processing thread corresponding to its change record writing partition, and the change log collection task is performed at the target node.

[0016] In a possible implementation manner of the first aspect, before the transaction is committed and before the execution data is sent to the target node and the message is registered, the method further includes:

[0017] Define agreed data formats;

[0018] Before the transaction is committed, the execution data is sent to the target node and the message is registered, including:

[0019] Before the transaction is committed, the change information of the execution data is encapsulated in the agreed data format;

[0020] Send the encapsulated execution data change information to the target node and register the message.

[0021] In a possible implementation manner of the first aspect, before the transaction is committed and before the execution data is sent to the target node and the message is registered, the method further includes:

[0022] According to business requirements, the target role attribute is granted to a node in the distributed system, the target node is obtained, and configuration information related to the target role attribute on all nodes in the distributed system is ensured to be synchronized.

[0023] In a possible implementation of the first aspect, in response to an instruction to enable change recording initiated by a user, before executing an operation that complies with a change recording rule on a working node and obtaining execution data, the method further includes:

[0024] Define change logging rules.

[0025] In a possible implementation manner of the first aspect, the change recording rules include library-level changes, schema-level changes, and object-level changes.

[0026] In a second aspect, an embodiment of the present application provides a data change log collection device in a distributed system, comprising:

[0027] An acquisition module, the acquisition module is used to respond to a change record start instruction initiated by a user, execute an operation that complies with the change record rule on a working node and obtain execution data;

[0028] A registration module, which is used to send execution data to a target node and register messages before a transaction is committed; wherein the target node is any node in a distributed system;

[0029] The collection module is used to perform a change log collection task at the target node in response to the target node receiving a transaction submission notification initiated by the working node when the transaction is submitted.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, the method for collecting data change logs in a distributed system as provided in any one of the first aspects above is implemented.

[0031] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to load and execute a computer program so that the electronic device executes the data change log collection method in a distributed system provided in any one of the first aspects above.

[0034] Compared with the prior art, the beneficial effects of this application are:

[0035] The embodiment of the present application proposes a data change log collection method, device, medium and equipment in a distributed system, the method comprising: in response to a change record start instruction initiated by a user, executing an operation that complies with the change record rules on a working node and obtaining execution data; before a transaction is committed, sending the execution data to a target node and registering a message; wherein the target node is any node in the distributed system; when a transaction is committed, in response to the target node receiving a transaction commit notification initiated by the working node, performing a change log collection task at the target node. The present application first responds to a change record start instruction initiated by a user, executes an operation that complies with the change record rules on a working node, and obtains corresponding execution data, and takes over the change record task through any node in the distributed system, thereby freeing up the resources occupied by the master node in the distributed system for maintaining the change record message, and there is no need to ensure that the change log data is dropped to disk before the transaction is committed. Since the target node will only start the change record task after receiving the transaction commit notification, it not only reduces the redundant processing caused by transaction rollback, but also solves the transaction delay problem caused by waiting for the change log to drop to disk before the transaction is committed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present application;

[0037] Figure 2 A flow chart of a method for collecting data change logs in a distributed system provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a target node collecting change record information from other nodes in a data change log collection method in a distributed system provided by an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the impact range of different change recording rules in the data change log collection method in a distributed system provided by an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a changed data registration structure in a data change log collection method in a distributed system provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a module of a data change log collection device in a distributed system provided in an embodiment of the present application;

[0042] Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] See attached Figure 1 , attached Figure 1 This is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present application. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or it may be a stable non-volatile memory (NVM), such as at least one disk storage; the processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or it may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0045] Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0046] As attached Figure 1 As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module, and a data change log collection device in a distributed system.

[0047] In the attached Figure 1In the electronic device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in the present application can be set in the electronic device, and the electronic device calls the data change log collection device in the distributed system stored in the memory 105 through the processor 101, and executes the data change log collection method in the distributed system provided in the embodiment of the present application.

[0048] Refer to the attached Figure 2 Based on the hardware device of the aforementioned embodiment, the embodiment of the present application provides a method for collecting data change logs in a distributed system, comprising the following steps:

[0049] S00: Define change recording rules.

[0050] In the specific implementation process, change record rules are defined to determine the operations that need to be performed on the working node. Change records support multiple rules with different granularity of impact scope, namely, library-level changes, schema-level changes, and object-level changes, which are respectively for all objects under the specified library, such as the attached Figure 4 The first line of instruction above indicates all objects in the specified mode, such as Figure 4 The second line of instruction indicates that the specified object is specified in the specified mode, as shown in the following figure. Figure 4 The data change operation represented by the third line of instructions above is recorded, which can better formulate a change record plan based on user needs and reasonably use system resources.

[0051] S10: In response to the change record start instruction initiated by the user, an operation that complies with the change record rule is executed on the working node and execution data is obtained.

[0052] In the specific implementation process, after agreeing on the change recording rules, the user opens the change recording in the form of instructions. All DDL or DML operations that comply with the change recording rules, that is, data definition language operations or data manipulation language operations, are executed on the working node. After the statement is executed, the working node obtains the execution data.

[0053] S20: before the transaction is committed, the execution data is sent to the target node and the message is registered; wherein the target node is any node in the distributed system.

[0054] In the specific implementation process, transaction commit is to commit all operations of the transaction, specifically, to write all updates to the database in the transaction back to the physical database on the disk, and the transaction ends normally. Before committing, a message is sent to the target node in the form of an internal cluster message for message registration. Since the target node is any node in the distributed system, it can be a working node other than the master node, freeing up the resources occupied by the master node in the distributed system for maintenance of change record messages, and there is no need to ensure that the change log data is written to the disk before the transaction is committed. The execution data is sent after the data is encapsulated, that is, before the transaction is committed, before the execution data is sent to the target node and the message is registered, the method also includes:

[0055] Define agreed data formats;

[0056] Before the transaction is committed, the execution data is sent to the target node and the message is registered, including:

[0057] Before the transaction is committed, the change information of the execution data is encapsulated in the agreed data format;

[0058] Send the encapsulated execution data change information to the target node and register the message.

[0059] The agreed data format is the format of data defined in advance. The data format is the specific arrangement and organization of data during storage or transmission, which determines how the data is represented and operated in the computer system. The types of data formats include text format, image format, database format, binary format, compression format, etc. After defining the agreed data format, the change information of the execution data can be encapsulated according to the corresponding encapsulation method.

[0060] In one embodiment, sending execution data to a target node and registering a message includes:

[0061] Send the execution data to the target node, and perform hash calculation on the transaction number to which the change message belongs based on the change message of the execution data to obtain the hash key value;

[0062] According to the hash key value, obtain the change record and write it into the partition;

[0063] Write the partition according to the change record and register the information.

[0064] In the specific implementation process, the target node will register the data change message after receiving it. The registration algorithm uses the Hash open chain method to hash the transaction number to which the change message belongs. The chain nodes with the same hash key value use the transaction number and the message source node number as the primary key information to avoid Hash collision. The number of partitions to write the change record will be set according to the change record partition parameter under each chain node, as shown in the attached figure. Figure 5 The figure shows a schematic diagram of the change data registration structure. The change message will decide in which partition to register based on the change positioning key of the message itself. The change positioning key is a general term for the fields in the internal row data structure of the change message that can participate in the partition positioning calculation. Only fields of comparable data types can be used as change positioning keys. The change positioning key can be a field or a combination of multiple fields.

[0065] In one embodiment, before the transaction is committed, before the execution data is sent to the target node and the message is registered, the method further includes:

[0066] According to business requirements, the target role attribute is granted to a node in the distributed system, the target node is obtained, and configuration information related to the target role attribute on all nodes in the distributed system is ensured to be synchronized.

[0067] In the specific implementation process, a target node role attribute is added in the embodiment of the present application. The role attribute can be granted to any node in the distributed system according to business needs. Once the target node is determined, other working nodes in the distributed system will also synchronize the target node information to reach a consensus to confirm the target node to which the data change message is sent. The target node collects change record messages from other nodes as shown in the attached Figure 3 As shown in the figure, the Master node is the cluster control node, the Query node is the working node, and the Gather node is the collection node, that is, the target node. The target role attribute supports the granting of attributes to any node in the distributed system. Users can implement load balancing between nodes based on the actual business workload.

[0068] S30: When the transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, a change log collection task is performed at the target node.

[0069] In the specific implementation process, the working node that initiates the transaction will register the change receipt information on the local transaction structure. The receipt information is used to send a transaction commit notification to the target node when the transaction is committed. The target node will start the change recording task only after receiving the transaction commit notification. This can not only reduce the redundant processing caused by transaction rollback, but also solve the transaction delay problem caused by waiting for the change log to be written to the disk before the transaction is committed. Transaction rollback is the database returning to the state at the beginning of the transaction: if a certain failure occurs during the operation of the transaction, the transaction cannot continue to execute, and the system will undo all completed update operations on the database in the transaction, so that the database rolls back to the state at the beginning of the transaction. And when the transaction is committed, the internal message of the cluster remotely notifies the target node to start the change recording task, which indirectly makes the change recording activity retain the atomicity of the transaction.

[0070] When the target node receives the remote transaction commit message, it will send the registered change message equal to the transaction number and the source node number of the message to the change message processing thread corresponding to each partition number for change recording, that is, to achieve multi-partition parallel writing of the change log, and solve the single point bottleneck problem that may exist in the change log writing performance. That is, when the transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, the change log collection task is performed at the target node, including:

[0071] When a transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, the target transaction number and the target message source node are obtained according to the hash key value;

[0072] The registered change message with the same target transaction number and target message source node is sent to the processing thread corresponding to its change record writing partition, and the change log collection task is performed at the target node.

[0073] In this embodiment, firstly, by responding to the change record start instruction initiated by the user, an operation that complies with the change record rules is executed on the working node, and the corresponding execution data is obtained, and the change record task is taken over by any node in the distributed system, thereby freeing up the resources occupied by the main node in the distributed system for maintaining the change record message, and there is no need to ensure that the change log data is written to the disk before the transaction is committed. Since the target node will only start the change record task after receiving the transaction submission notification, it not only reduces the redundant processing caused by transaction rollback, but also solves the transaction delay problem caused by waiting for the change log to be written to the disk before the transaction is committed.

[0074] See attached Figure 6 Based on the same inventive concept as in the above-mentioned embodiment, the embodiment of the present application further provides a data change log collection device in a distributed system, comprising:

[0075] An acquisition module, the acquisition module is used to respond to a change record start instruction initiated by a user, execute an operation that complies with the change record rule on a working node and obtain execution data;

[0076] A registration module, which is used to send execution data to a target node and register messages before a transaction is committed; wherein the target node is any node in a distributed system;

[0077] The collection module is used to perform a change log collection task at the target node in response to the target node receiving a transaction submission notification initiated by the working node when the transaction is submitted.

[0078] Those skilled in the art should understand that the division of the various modules in the embodiment is merely a division of logical functions, and in actual application, all or part of them can be integrated into one or more actual carriers, and these modules can be implemented entirely in the form of software called by a processing unit, or entirely in the form of hardware, or in the form of a combination of software and hardware. It should be noted that each module in the data change log collection device under a distributed system in this embodiment corresponds one-to-one to each step in the data change log collection method under a distributed system in the aforementioned embodiment. Therefore, the specific implementation method of this embodiment can refer to the implementation method of the data change log collection method under a distributed system, which will not be repeated here.

[0079] Based on the same inventive concept as in the aforementioned embodiment, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, a data change log collection method in a distributed system as provided in an embodiment of the present application is implemented.

[0080] Based on the same inventive concept as in the aforementioned embodiments, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program; the processor is used to load and execute the computer program so that the electronic device executes a data change log collection method in a distributed system as provided in an embodiment of the present application.

[0081] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0082] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0083] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0084] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0086] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0088] In summary, the present application provides a method, device, medium and equipment for collecting data change logs in a distributed system, the method comprising: in response to a user-initiated instruction to start change recording, executing operations that comply with change recording rules on a working node and obtaining execution data; before a transaction is committed, sending the execution data to a target node and registering a message; wherein the target node is any node in the distributed system; when a transaction is committed, in response to the target node receiving a transaction submission notification initiated by a working node, performing a change log collection task at the target node. The present application first responds to a user-initiated instruction to start change recording, executes operations that comply with change recording rules on a working node, and obtains corresponding execution data, and takes over the change recording task through any node in the distributed system, freeing up the resources occupied by the master node in the distributed system for maintaining change recording messages, and there is no need to ensure that the change log data is dropped to disk before the transaction is committed. Since the target node will only start the change recording task after receiving the transaction submission notification, it not only reduces the redundant processing caused by transaction rollback, but also solves the transaction delay problem caused by waiting for the change log to drop to disk before the transaction is committed.

[0089] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for collecting data change logs in a distributed system, characterized in that: The following steps are involved: In response to a change record start instruction initiated by a user, an operation that complies with the change record rule is executed on a working node and execution data is obtained; Before the transaction is committed, the execution data is sent to the target node and the message is registered; wherein the target node is any node in the distributed system; When the transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, a change log collection task is performed at the target node.

2. The method for collecting data change logs in a distributed system according to claim 1, characterized in that: The sending of the execution data to the target node and registering the message includes: The execution data is sent to the target node, and according to the change message of the execution data, a hash calculation is performed on the transaction number to which the change message belongs to obtain a hash key value; According to the hash key value, obtain the change record and write it into the partition; The change record is written into the partition and the message is registered.

3. The method for collecting data change logs in a distributed system according to claim 2, characterized in that: When the transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, performing a change log collection task at the target node, including: When the transaction is committed, in response to the target node receiving the transaction commit notification initiated by the working node, obtaining the target transaction number and the target message source node according to the hash key value; The registered change message having the same target transaction number and target message source node is sent to the processing thread corresponding to the change record writing partition, and the change log collection task is performed at the target node.

4. The method for collecting data change logs in a distributed system according to claim 1, characterized in that: Before the transaction is committed, before the execution data is sent to the target node and the message is registered, the method further includes: Define agreed data formats; Before the transaction is committed, the execution data is sent to the target node and message registration is performed, including: Before the transaction is committed, encapsulating the change information of the execution data in the agreed data format; The encapsulated change information of the execution data is sent to the target node and the message is registered.

5. The method for collecting data change logs in a distributed system according to claim 1, characterized in that: Before the transaction is committed, before the execution data is sent to the target node and the message is registered, the method further includes: According to business requirements, a target role attribute is granted to a node in the distributed system, the target node is obtained, and configuration information related to the target role attribute on all nodes in the distributed system is ensured to be synchronized.

6. The method for collecting data change logs in a distributed system according to claim 1, characterized in that: In response to the change record start instruction initiated by the user, before executing the operation that complies with the change record rule on the working node and obtaining the execution data, the method further includes: The change recording rules are defined.

7. The method for collecting data change logs in a distributed system according to claim 6, characterized in that: The change recording rules include library-level changes, schema-level changes, and object-level changes.

8. A data change log collection device in a distributed system, characterized in that: include: An acquisition module, the acquisition module is used to respond to the change record start instruction initiated by the user, execute an operation that complies with the change record rule on the working node and obtain execution data; A registration module, the registration module is used to send the execution data to a target node and perform message registration before the transaction is committed; wherein the target node is any node in the distributed system; A collection module, wherein the collection module is used to perform a change log collection task at the target node in response to the target node receiving a transaction submission notification initiated by the working node when the transaction is submitted.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by the processor, the data change log collection method in a distributed system according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the data change log collection method in a distributed system according to any one of claims 1 to 7.

Images