Database capturing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a database capturing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a data change capture request; and in response to the data change capture request, establishing a logic copy stream with a target database through the transformed database capture plug-in, and capturing a real-time change log of the target database based on the logic copy stream. By adopting the method, the change events of the heterogeneous database can be uniformly managed under the same framework.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database crawling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Against the backdrop of the promotion of domestically developed information technology databases, more and more business systems are migrating to GaussDB databases. In order to support business scenarios such as real-time risk control and real-time analysis, the system usually needs to use CDC (Data Change Detection and Control) to synchronize data changes in the database to downstream message middleware and various real-time processing systems in real time.

[0003] Some Debezium components are widely used for change capture of databases such as MySQL and PostgreSQL, but they do not natively support GaussDB, which makes it impossible to uniformly manage change events of heterogeneous databases under the same framework. Summary of the Invention

[0004] Therefore, it is necessary to provide a database crawling method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can uniformly manage the database, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a database crawling method, including:

[0006] Receive data change capture requests;

[0007] In response to the data change capture request, a logical replication stream is established with the target database through the modified database capture plugin, and the real-time change log of the target database is captured based on the logical replication stream.

[0008] In one embodiment, establishing a logical replication stream with the target database through the modified database capture plugin includes:

[0009] The modified database capture plugin calls the logical replication interface in the target database to create a logical replication slot;

[0010] The modified database capture plugin establishes the logical replication flow based on the logical replication slot.

[0011] In one embodiment, before creating the logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin, the method further includes:

[0012] The modified database capture plugin establishes an initial connection with the target database and completes the connection configuration by combining preset connection parameters.

[0013] In one embodiment, the step of capturing the real-time change logs of the target database based on the logical replication stream includes:

[0014] The modified database capture plugin receives change data output from the logic decoding plugin of the target database based on the logic replication stream;

[0015] The modified database capture plugin performs semantic parsing and field extraction on the changed data to obtain the change log.

[0016] In one embodiment, the modification process of the database capture plugin includes:

[0017] Obtain the raw database capture plugin, and determine the target functional part from the raw database capture plugin;

[0018] The target functional part is modified by one or more of the following methods: logical copy interface modification and data parsing modification, to obtain the modified database capture plugin.

[0019] In one embodiment, the logical replication interface is modified by changing the instruction calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the modified data parsing is modified by adding a parsing processor to the original database capture plugin for parsing the output format of the logical decoding plugin.

[0020] Secondly, this application also provides a database crawling device, comprising:

[0021] The receiving module is used to receive data change capture requests;

[0022] The capture module is used to respond to the data change capture request, establish a logical replication stream with the target database through the modified database capture plugin, and capture the real-time change log of the target database based on the logical replication stream.

[0023] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0024] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0025] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0026] The aforementioned database capture method, apparatus, computer equipment, computer-readable storage medium, and computer program product receive data change capture requests, then respond to the requests by establishing a logical replication stream with the target database through a modified database capture plugin, and captures the target database's real-time change logs based on the plugin. Because the modified database capture plugin can adapt to the target database's logical decoding output and convert changed data into change events in a unified format, change events from heterogeneous databases can be managed uniformly within the same framework. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a database crawling method in one embodiment;

[0029] Figure 2 This is a schematic diagram of a data scraping method in an exemplary embodiment;

[0030] Figure 3 This is a structural block diagram of a data capture device in one embodiment;

[0031] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] In one embodiment, such as Figure 1 As shown, a database crawling method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0034] Step 102: Receive data change capture request.

[0035] Among them, the data change capture request refers to the process used to instruct the system to capture data changes to the target database.

[0036] For example, when the system detects that the target database configuration is complete or the upper-level system triggers a change capture task, a data change capture request can be generated.

[0037] Step 104: In response to the data change capture request, establish a logical replication stream with the target database through the modified database capture plugin, and capture the real-time change log of the target database based on the logical replication stream.

[0038] Among them, the logical replication stream pre-creates and establishes a long-term connection channel to continuously receive change logs output by the logical decoding plugin when changes occur in the target database.

[0039] Specifically, after receiving a data change capture request, the system uses the modified database acquisition plugin to establish a logical replication stream with the target database, and captures the real-time change logs of the target database in real time through the logical replication stream.

[0040] The target database refers to the data source that can output database change logs, which usually includes database instances that support logical replication or equivalent data change output mechanisms, such as the GaussDB database that supports WAL logical decoding. The modified database capture plugin refers to the data change capture component obtained by enhancing the functionality of the original database capture plugin. It is used to establish a logical replication stream with the target database and parse the change data output by logical decoding. For example, a plugin obtained by modifying the PostgreSQL capture plugin based on Debezium.

[0041] Since the original database capture plugin cannot directly use the logical replication API to create logical replication slots and cannot parse the change data format output by the target database logical decoding plugin, it can be modified to support the acquisition of change data based on logical replication slots and the parsing of custom decoding formats, thereby adapting to the data change output mechanism of the target database.

[0042] For example, the original database capture plugin could be Debezium (Debezium Change DataCapture Framework), which is used to capture changed data from various databases based on log parsing. This plugin is suitable for databases that support standard output formats and general logical decoding methods, but it does not support creating replication slots via the logical replication API, nor can it parse changed data output in a specific logical decoding plugin (e.g., MPPDB_DECODING). Therefore, this embodiment modifies Debezium to enable it to create replication slots based on the logical replication API and support parsing changed data in a specific decoding format.

[0043] In the above embodiment, a data change capture request is received, and then a logical replication stream is established with the target database through the modified database capture plugin in response to the request. The plugin then captures the real-time change logs of the target database. Since the modified database capture plugin can adapt to the logical decoding output of the target database and convert the changed data into change events in a unified format, change events from heterogeneous databases can be managed uniformly within the same framework.

[0044] In one embodiment, the above-mentioned establishment of a logical replication flow with the target database through the modified database capture plugin includes: creating a logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin; and establishing a logical replication flow based on the logical replication slot through the modified database capture plugin.

[0045] The logical replication interface refers to the interface provided by the target database for managing logical replication resources. It is used to perform operations such as creating logical replication slots, querying slot status, and managing replication streams. A logical replication slot refers to a logical replication configuration information persistently stored in the target database. This information is used to record the read position of the change log and prevent the related WAL (Write-Ahead Logging) from being prematurely reclaimed. In other words, the database reserves a storage location for the WAL logs for change subscribers to prevent log deletion and record consumption progress.

[0046] Optionally, when calling the logical replication interface to create a logical replication slot, the modified database capture plugin replaces the original SQL command-based creation method with a logical replication API-based creation method, and specifies the logical decoding plugin and slot name during creation to generate a logical replication slot that can be used to output logically decoded data.

[0047] Optionally, the modified database capture plugin establishes a logical replication stream based on the logical replication slot, including: initiating a replication stream request based on the logical replication API and passing the logical replication slot as a parameter, so that the target database can push the logical decoding results in real time through the replication stream.

[0048] For example, the logical replication interface is used to call the replication API provided by the database to create logical replication slots and pass them through the replication stream; after the replication stream passes through, a logical replication stream is formed, which continuously reads the logically decoded change data based on the consumption position recorded in the replication slot.

[0049] In the above embodiments, by creating logical replication slots through the logical replication interface and continuously reading decoding logs through logical replication streams, the dependence on specific SQL implementations can be reduced, and the parsing of different databases or different logical decoding plugin output formats can be supported, thereby improving the system's adaptability to heterogeneous databases.

[0050] In one embodiment, before creating a logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin, the method further includes: establishing an initial connection with the target database through the modified database capture plugin, and completing the connection configuration in combination with preset connection parameters.

[0051] The initial connection refers to the basic communication connection established between the database connection driver and the target database, serving as the starting point for the session to perform logical replication-related operations. For example, in an implementation based on the PostgreSQL protocol, the basic connection can be obtained through a JDBC driver. Preset connection parameters refer to the connection attributes used to configure this initial connection, including but not limited to setting the query mode to simple mode to ensure that subsequent logical replication API calls can be made correctly.

[0052] Specifically, in this embodiment, the modified database capture plugin establishes a basic connection with the target database through a JDBC driver, and sets the query mode parameter to simple in the connection configuration when establishing the connection; after the initial connection is established, the basic connection is transformed into a logical replication interface (e.g., ReplicationConnection) so that logical replication slots can be created and logical replication streams can be passed through the logical replication interface.

[0053] In the above embodiments, by explicitly setting the query mode to simple mode when establishing the initial connection, and calling the logical replication API to create a logical replication slot based on the initial connection and through replication stream, the dependency on database syntax for creating logical replication slots through SQL commands can be avoided.

[0054] In one embodiment, the above-mentioned capture of real-time change logs of the target database based on logical replication stream includes: receiving change data output from the logical decoding plugin of the target database based on logical replication stream through the modified database capture plugin; and performing semantic parsing and field extraction on the change data through the modified database capture plugin to obtain change logs.

[0055] Specifically, the database uses a logical decoding plugin to convert WAL logs into consumable change data. The improved database capture plugin continuously pulls this change data based on a logical replication stream and performs semantic parsing and field extraction on the change data to obtain a structured change log.

[0056] For example, when a row of data in the target database is updated, the logic decoding plugin parses the corresponding WAL log into logical change data and outputs it to the database capture plugin via a logical replication stream. The database capture plugin receives the change record based on the logical replication stream, performs structured parsing according to table name, field name, and the old and new values ​​of the fields, and outputs the parsing result as a change log for downstream systems to use for data synchronization or event processing.

[0057] In the above embodiments, by obtaining logically decoded change data from the target database based on logical replication streams and performing semantic parsing on the change data, real-time incremental capture can be achieved without polling the business tables, reducing the database read load; at the same time, the decoded data is in a structured format, which is convenient for downstream systems to process directly.

[0058] In one embodiment, the modification process of the database capture plugin includes: obtaining the original database capture plugin, determining the target functional part from the original database capture plugin, and modifying the target functional part by one or more of the following: logical replication interface modification and data parsing modification, to obtain the modified database capture plugin.

[0059] The target functionality refers to the set of functional modules in the original database capture plugin used to create logical replication slots, establish replication streams, and parse changed data. For example, when the original database capture plugin is the PostgreSQL capture plugin in the Debezium data change capture framework, the code responsible for executing SQL commands to create replication slots, establishing logical replication streams, and parsing logical decoded output data can be considered as the target functionality.

[0060] After determining the target functional part, one or more of the following modifications are performed on the target functional part: logical replication interface modification and change data parsing modification. This enables the modified database capture plugin to create logical replication slots through the logical replication interface and to parse the change data format output by the target database logical decoding plugin, thereby adapting to the data change capture requirements of the target database.

[0061] Furthermore, the aforementioned logical replication interface was modified by changing the command calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the data parsing was modified by adding a parsing processor to the original database capture plugin to parse the output format of the logical decoding plugin.

[0062] For example, modifying the logical replication interface may include: deleting the code path in the original database capture plugin that creates logical replication slots by executing SQL statements such as generating logical replication streams; instead, obtaining a basic connection through a database connection driver, converting the basic connection into a logical replication interface, and calling the create replication slot method provided by the logical replication interface to create the logical replication slot. When creating the logical replication slot, the slot name and the logical decoding plugin name are passed as parameters, enabling the logical replication slot to output changed data in the format of the target logical decoding plugin.

[0063] For example, the above-mentioned modification to change data parsing may include: adding a parsing processor module for the target logic decoding plugin to the original database capture plugin, such as adding a parsing class for processing the output format; in the parsing processor, the change record is parsed in segments according to the text or binary format output by the logic decoding plugin, extracting information such as table name, operation type, field name, field type, and old and new values ​​of the fields, and assembling the parsing results into a structured change event object; at the same time, during the plugin initialization process, an association is established between the parsing processor and the name of the target logic decoding plugin, so that when it is detected that the current logic replication slot uses the logic decoding plugin, the corresponding parsing processor can be automatically selected to parse the change data.

[0064] In the above embodiments, by modifying the logical replication interface and change data parsing of the original database capture plugin, the database capture plugin can create logical replication slots and consume change data output by the target logical decoding plugin in the form of a logical replication interface, without relying on specific SQL commands or default decoding formats. At the same time, by adding a data parsing processor for the target logical decoding plugin, the decoding results can be parsed into structured change events, which are convenient for downstream systems to consume directly, reducing access costs and improving data processing efficiency.

[0065] In one embodiment, combined Figure 2 , Figure 2This is a schematic diagram of a data scraping method in an exemplary embodiment.

[0066] First, a Replication Slot (the logical replication slot in the above embodiment) is created using the modified database capture plugin. This slot is then used to record the consumption location of the change log and specify the logical decoding plugin. Next, based on the successful creation of the logical replication slot, the modified database capture plugin calls the logical replication interface through a logical replication stream to establish a logical replication stream and continuously receive change data output by the logical decoding plugin from the target database. Subsequently, semantic parsing and field extraction are performed on the change data to obtain a structured change log.

[0067] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0068] Based on the same inventive concept, this application also provides a database crawling device for implementing the database crawling method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more database crawling device embodiments provided below can be found in the limitations of the database crawling method described above, and will not be repeated here.

[0069] In one exemplary embodiment, such as Figure 3 As shown, a database crawling device is provided, including: a receiving module and a crawling module, wherein:

[0070] The receiving module is used to receive data change capture requests.

[0071] The capture module is used to respond to data change capture requests. It establishes a logical replication stream with the target database through the modified database capture plugin and captures the real-time change logs of the target database based on the logical replication stream.

[0072] In one embodiment, the above-mentioned crawling module includes:

[0073] The replication slot creation unit is used to create logical replication slots by calling the logical replication interface in the target database through the modified database capture plugin.

[0074] The replication stream creation unit is used to create a logical replication stream based on the logical replication slot through the modified database capture plugin.

[0075] In one embodiment, the above-mentioned crawling module further includes:

[0076] The configuration unit is used to establish an initial connection with the target database through the modified database capture plugin and complete the connection configuration in combination with preset connection parameters.

[0077] In one embodiment, the above-mentioned crawling module includes:

[0078] The data receiving unit is used to receive change data output from the logical decoding plugin of the target database based on the logical replication stream through the modified database capture plugin.

[0079] The parsing unit is used to perform semantic parsing and field extraction on the changed data through the modified database capture plugin to obtain the change log.

[0080] In one embodiment, the above-mentioned apparatus includes:

[0081] The acquisition module is used to acquire the raw database capture plugin and determine the target functional part from the raw database capture plugin.

[0082] The modification module is used to modify one or more of the following: logical copy interface modification and data parsing modification of the target function part, to obtain the modified database capture plugin.

[0083] In the above embodiments, the logical replication interface in the above-mentioned modification module is modified to change the instruction calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the change data parsing in the above-mentioned modification module is modified to add a parsing processor to the original database capture plugin for parsing the output format of the logical decoding plugin.

[0084] Each module in the aforementioned database crawling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0085] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores changed data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a database retrieval method.

[0086] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0087] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: receiving a data change capture request; in response to the data change capture request, establishing a logical replication stream with a target database through a modified database capture plugin, and capturing real-time change logs of the target database based on the logical replication stream.

[0088] In one embodiment, when the processor executes the computer program, it also performs the following steps: creating a logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin; and establishing a logical replication flow based on the logical replication slot through the modified database capture plugin.

[0089] In one embodiment, when the processor executes the computer program, it also performs the following steps: establishing an initial connection with the target database through the modified database capture plugin, and completing the connection configuration in combination with preset connection parameters.

[0090] In one embodiment, when the processor executes the computer program, it also performs the following steps: receiving change data output from the logical decoding plugin of the target database based on the logical replication stream through the modified database capture plugin; and performing semantic parsing and field extraction on the change data through the modified database capture plugin to obtain a change log.

[0091] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the original database capture plugin, determining the target functional part from the original database capture plugin, and performing one or more of the following on the target functional part: logical copy interface modification and data parsing modification, to obtain the modified database capture plugin.

[0092] In one embodiment, when the processor executes a computer program, the aforementioned logical replication interface is modified to change the instruction calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the data parsing is modified to add a parsing processor to the original database capture plugin for parsing the output format of the logical decoding plugin.

[0093] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: receiving a data change capture request; in response to the data change capture request, establishing a logical replication stream with a target database through a modified database capture plugin, and capturing real-time change logs of the target database based on the logical replication stream.

[0094] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: creating a logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin; and establishing a logical replication flow based on the logical replication slot through the modified database capture plugin.

[0095] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: establishing an initial connection with the target database through the modified database capture plugin, and completing the connection configuration in combination with preset connection parameters.

[0096] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving change data output from the logical decoding plugin of the target database based on the logical replication stream through the modified database capture plugin; and performing semantic parsing and field extraction on the change data through the modified database capture plugin to obtain a change log.

[0097] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the original database capture plugin, determining the target functional part from the original database capture plugin, and performing one or more of the following on the target functional part: logical copy interface modification and data parsing modification, to obtain the modified database capture plugin.

[0098] In one embodiment, when the computer program is executed by the processor, the above-mentioned logical replication interface is modified to change the instruction calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the data parsing is modified to add a parsing processor to the original database capture plugin for parsing the output format of the logical decoding plugin.

[0099] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: receiving a data change capture request; in response to the data change capture request, establishing a logical replication stream with a target database through a modified database capture plugin, and capturing real-time change logs of the target database based on the logical replication stream.

[0100] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: creating a logical replication slot by calling the logical replication interface in the target database through the modified database capture plugin; and establishing a logical replication flow based on the logical replication slot through the modified database capture plugin.

[0101] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: establishing an initial connection with the target database through the modified database capture plugin, and completing the connection configuration in combination with preset connection parameters.

[0102] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving change data output from the logical decoding plugin of the target database based on the logical replication stream through the modified database capture plugin; and performing semantic parsing and field extraction on the change data through the modified database capture plugin to obtain a change log.

[0103] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the original database capture plugin, determining the target functional part from the original database capture plugin, and performing one or more of the following on the target functional part: logical copy interface modification and data parsing modification, to obtain the modified database capture plugin.

[0104] In one embodiment, when the computer program is executed by the processor, the above-mentioned logical replication interface is modified to change the instruction calling method used to create logical replication slots in the original database capture plugin to a creation method based on the logical replication interface; the data parsing is modified to add a parsing processor to the original database capture plugin for parsing the output format of the logical decoding plugin.

[0105] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0107] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A database scraping method, characterized by, The method comprises: receiving a data change capture request; in response to the data change capture request, establishing a logical replication stream by a modified database capture plug-in and a target database, and capturing a real-time change log of the target database based on the logical replication stream.

2. The method of claim 1, wherein, The establishment of the logical replication stream by the modified database capture plug-in and the target database comprises: creating a logical replication slot by the modified database capture plug-in calling a logical replication interface in the target database; establishing the logical replication stream based on the logical replication slot by the modified database capture plug-in.

3. The method of claim 2, wherein, Before the creation of the logical replication slot by the modified database capture plug-in calling the logical replication interface in the target database, the method further comprises: establishing an initial connection by the modified database capture plug-in and the target database, and completing connection configuration in combination with preset connection parameters.

4. The method of claim 1, wherein, The capturing of the real-time change log of the target database based on the logical replication stream comprises: receiving change data output from a logical decoding plug-in of the target database based on the logical replication stream by the modified database capture plug-in; performing semantic analysis and field extraction on the change data by the modified database capture plug-in to obtain the change log.

5. The method of claim 1, wherein, The modification process of the database capture plug-in comprises: obtaining an original database capture plug-in, determining a target function part from the original database capture plug-in; performing one or more of logical replication interface modification and change data analysis modification on the target function part to obtain the modified database capture plug-in.

6. The method of claim 5, wherein, The logical replication interface modification is to adjust the instruction calling mode for creating a logical replication slot in the original database capture plug-in to a creation mode based on a logical replication interface; and the change data analysis modification is to add an analysis processor for analyzing the output format of a logical decoding plug-in in the original database capture plug-in.

7. A database scraping apparatus, characterized by, The apparatus comprises: a receiving module configured to receive a data change capture request; a capturing module configured to, in response to the data change capture request, establish a logical replication stream by a modified database capture plug-in and a target database, and capture a real-time change log of the target database based on the logical replication stream.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 6 when executed by the processor.

10. A computer program product comprising a computer program, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 6 when executed by the processor. The computer program implements the steps of the method of any one of claims 1 to 6 when executed by the processor.

Images