A distributed database change data retrieval method, device, equipment and medium

Abstract

The embodiment of the present invention discloses a distributed database change data retrieval method, device, equipment and medium. The method includes: obtaining retrieval configuration information corresponding to the target retrieval task; determining the target retrieval result corresponding to the target retrieval task based on the full logical log and retrieval configuration information of the distributed database; determining the target retrieval file corresponding to the target retrieval task based on the preset output format, the current task state corresponding to the target retrieval task and the target retrieval result. Through the technical solution of the embodiment of the present invention, the retrieval task can be completed conveniently and accurately and the target retrieval file corresponding to the retrieval task can be obtained, which realizes one-click retrieval of change data and improves the efficiency and accuracy of change data retrieval.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of database technology, and in particular to a distributed database change data retrieval method, device, equipment and medium. Background Art

[0002] In the process of database business operation, the traceability of data changes is very important. It is often necessary to retrieve the data changes in the business database in the past period of time, and to find out how a row of data was updated.

[0003] Currently, to retrieve data changes, people need to manually use relevant database tools to filter a large amount of data in the business database to obtain the changed data, and then query the log based on the filtered changed data to obtain the specific data change operations. However, this manual retrieval method is highly dependent on human experience, which reduces the efficiency and accuracy of retrieval of changed data and data change operations. Summary of the invention

[0004] The embodiments of the present invention provide a distributed database change data retrieval method, device, equipment and medium to conveniently and accurately complete the retrieval task and obtain the target retrieval file corresponding to the retrieval task, thereby realizing one-click retrieval of change data and improving the efficiency and accuracy of change data retrieval.

[0005] In a first aspect, an embodiment of the present invention provides a distributed database change data retrieval method, comprising:

[0006] Obtain the retrieval configuration information corresponding to the target retrieval task;

[0007] Determine the target search result corresponding to the target search task based on the full logical log of the distributed database and the search configuration information;

[0008] Based on a preset output format, a current task state corresponding to the target retrieval task and the target retrieval result, a target retrieval file corresponding to the target retrieval task is determined.

[0009] Optionally, the method also includes: the retrieval configuration information includes: first retrieval configuration information and second retrieval configuration information; based on the full logical log of the distributed database and the first retrieval configuration information, determining the target data change log corresponding to the target retrieval task; parsing the target data change log to determine the target resolution information corresponding to the target retrieval task; based on the second retrieval configuration information and the target resolution information, determining the target retrieval result corresponding to the target retrieval task.

[0010] Optionally, the method also includes: the retrieval configuration information is table dimension retrieval information; wherein the table dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier, a library table name, a specified column filter condition, an action type, and a memory limit during the retrieval process; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier, a library table name, a specified column filter condition, an action type, and a memory limit during the retrieval process.

[0011] Optionally, the method also includes: the retrieval configuration information is transaction dimension retrieval information; wherein the transaction dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier and a transaction size; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier and a transaction size.

[0012] Optionally, the method further includes: if the current task status corresponding to the target retrieval task is a completed status, determining the target retrieval file corresponding to the target retrieval task based on a preset output format and the target retrieval result corresponding to the target retrieval task.

[0013] Optionally, the method further includes: if the current task status corresponding to the target retrieval task is a completed status, determining the target retrieval file corresponding to the target retrieval task based on a preset output format and the target retrieval result corresponding to the target retrieval task.

[0014] Optionally, the method further includes: detecting a current subtask state corresponding to each subtask in a distributed database, and determining a current task state corresponding to the target retrieval task based on the current subtask state corresponding to each subtask; wherein the subtask corresponds to a shard identifier.

[0015] In a second aspect, an embodiment of the present invention further provides a distributed database change data retrieval device, the device comprising:

[0016] A retrieval configuration information acquisition module is used to obtain the retrieval configuration information corresponding to the target retrieval task;

[0017] A target search result determination module, used to determine the target search result corresponding to the target search task based on the full logical log of the distributed database and the search configuration information;

[0018] The target retrieval file determination module is used to determine the target retrieval file corresponding to the target retrieval task based on a preset output format, a current task state corresponding to the target retrieval task and the target retrieval result.

[0019] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0020] one or more processors;

[0021] A memory for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed database change data retrieval method provided by any embodiment of the present invention.

[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a distributed database change data retrieval method as provided in any embodiment of the present invention.

[0024] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements a distributed database change data retrieval method as provided in any embodiment of the present invention.

[0025] The technical solution of the embodiment of the present invention obtains the retrieval configuration information corresponding to the target retrieval task; determines the target retrieval result corresponding to the target retrieval task based on the full logical log of the distributed database and the retrieval configuration information; determines the target retrieval file corresponding to the target retrieval task based on a preset output format, the current task status corresponding to the target retrieval task and the target retrieval result, thereby realizing one-click retrieval of changed data without human intervention, thereby improving the efficiency and accuracy of changed data retrieval.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 It is a flow chart of a distributed database change data retrieval method provided by the first embodiment of the present invention;

[0029] Figure 2is a flow chart of a distributed database change data retrieval method provided by Embodiment 2 of the present invention;

[0030] Figure 3 It is a structural diagram of a distributed database change data retrieval device provided by Embodiment 3 of the present invention;

[0031] Figure 4 It is a structural schematic diagram of an electronic device for implementing the distributed database change data retrieval method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a distributed database change data retrieval method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of one-click retrieval of distributed database change data. The method can be executed by a distributed database change data retrieval device. The distributed database change data retrieval device can be implemented in the form of hardware and / or software. The distributed database change data retrieval device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0036] S110: Obtain search configuration information corresponding to the target search task.

[0037] The target search task may refer to a task of searching for changed data and data change operations in a database. The search configuration information may refer to pre-configured search screening conditions.

[0038] Specifically, the retrieval configuration information pre-configured by the intelligent operation and maintenance system (Insight) of the distributed database cluster for the target retrieval task is obtained.

[0039] S120: Determine a target search result corresponding to the target search task based on the full logical log and search configuration information of the distributed database.

[0040] Among them, a distributed database may refer to a database system in which data is dispersedly stored in multiple physical locations. These locations may be multiple geographically dispersed servers, data centers, or cloud platforms. Users access these databases through the network. Distributed databases are different from traditional centralized databases. The data of centralized databases is stored on a single server, facing problems such as single point failure and performance bottlenecks. The full logical log may refer to all logical logs in a distributed database. A logical log may refer to a log that records database modification operations. The target retrieval result may refer to the log parsing information obtained by filtering based on the retrieval configuration information. The target retrieval result can be used to generate a target retrieval file.

[0041] Specifically, the full logical log of the distributed database is parsed to obtain full log parsing information, and the full log parsing information is screened using the retrieval configuration information to obtain a target retrieval result corresponding to the target retrieval task.

[0042] S130: Determine a target retrieval file corresponding to the target retrieval task based on a preset output format, a current task status corresponding to the target retrieval task, and a target retrieval result.

[0043] The preset output format may refer to a preset format for generating a target search file. The current task status may refer to the task status of the target search task at the current moment. The current task status may include: a completed state and an uncompleted state. The target search file may refer to a search result file displayed to the configuration party of the search configuration information.

[0044] Specifically, if the current task state corresponding to the target retrieval task is an unfinished state, the current task state is continuously detected. If the current task state corresponding to the target retrieval task is a completed state, the target retrieval result corresponding to the target retrieval task is generated into a target retrieval file corresponding to the target retrieval task according to a preset output format.

[0045] On the basis of the above technical solution, "determining the target retrieval file corresponding to the target retrieval task based on the preset output format, the current task status corresponding to the target retrieval task and the target retrieval result" may include: if the current task status corresponding to the target retrieval task is a completed state, then determining the target retrieval file corresponding to the target retrieval task based on the preset output format and the target retrieval result corresponding to the target retrieval task.

[0046] Among them, the preset output format may include: the identifier of the global transaction coordination center (GTMGtid), the identifier of the data node (DNGtid), the execution time, the log file identifier (logFile), the position information of a specific operation in the logical log file (LogPOS), and the forward structured query language (sql) statement.

[0047] Specifically, if the current task status corresponding to the target retrieval task is completed, the target retrieval result corresponding to the target retrieval task is generated into a target retrieval file corresponding to the target retrieval task according to the preset output format. For example, in the target retrieval file, GTMGtid is 0; DNGtid is 2b3ff15f-d15d-11ee-bf78-287b09cb3c0e:37; execution time is 2024-02-2216:52:19; logFile is data / logfile / bin.000003; LogPOS is 13546; and the forward SQL statement is INSERT INTO t1.test(`id`,`GDB_BID`,`GTID`)VALUES(1,218266,0).

[0048] Based on the above technical solution, the method also includes: before determining the target retrieval file, detecting the current subtask status corresponding to each subtask in the distributed database, and determining the current task status corresponding to the target retrieval task based on the current subtask status corresponding to each subtask.

[0049] Among them, the subtask corresponds to the shard identifier. If there is a shard identifier in the retrieval configuration information, the target retrieval task contains multiple subtasks. A retrieval task on a shard corresponding to a shard identifier is a subtask. The current subtask status may refer to the task status of the subtask in the target retrieval task at the current moment. Database sharding is a database management technology. Its core idea is to split a large database into multiple small database instances, each of which is called a shard. Each shard contains a portion of the database data and the corresponding index, and can store and process data independently. The purpose of sharding is to improve the performance and scalability of the database. By distributing the load to multiple servers, large amounts of data and requests can be processed more efficiently. The advantage of this setting is that in a distributed database multi-shard scenario, it can avoid the data center operation and maintenance personnel from spending a lot of energy to parse logical logs and confirm the current task status, and support one-click retrieval of data changes in multiple dimensions.

[0050] Specifically, the current subtask status corresponding to each subtask in the distributed database is detected in real time. If there is at least one unfinished state in the current subtask status corresponding to all subtasks, the current task status corresponding to the target retrieval task is an unfinished state, and the main task status in the table record is maintained as in progress. If the current subtask status corresponding to all subtasks is a completed state, the current task status corresponding to the target retrieval task is determined to be a completed state, and the main task status in the table record is updated to completed.

[0051] The technical solution of the embodiment of the present invention obtains the retrieval configuration information corresponding to the target retrieval task; determines the target retrieval result corresponding to the target retrieval task based on the full logical log and retrieval configuration information of the distributed database; determines the target retrieval file corresponding to the target retrieval task based on a preset output format, the current task status corresponding to the target retrieval task and the target retrieval result, thereby realizing one-click retrieval of changed data without human intervention, thereby improving the efficiency and accuracy of changed data retrieval.

[0052] It should be noted that the embodiment of the present invention also provides a one-click search system, including: a parameter configuration module, a search implementation module and a result viewing module. The functions of the parameter configuration module and the result viewing module can be implemented by the intelligent operation and maintenance system (Insight) of the distributed database cluster. The function of the search implementation module can be implemented by a database agent (DBAgent).

[0053] Among them, Insight plays the role of parameter configuration and distribution, and the page displays the results and provides the result download function, so that users can download the data changes to the local for viewing and analysis. The multiple parameters used for screening involved in the present invention are designed as optional and configurable configuration items. Users can flexibly configure the data changes of the specified time range, specified shards, specified libraries and tables, specified column changes, specified action types, and specified transaction sizes according to the target retrieval operation statement, and can limit the size of the output file, further improving the flexibility of the change data retrieval.

[0054] DBAgent is used to monitor and start and stop the daemon process of the database instance. The parsing and filtering of logical logs are completed by DBAgent, and DBAgent returns the results to Insight.

[0055] Specifically, Insight configures the retrieval configuration information corresponding to the target retrieval task and sends it to DBAgent. DBAgent is used to execute steps S110 to S130. When the current task status corresponding to the target retrieval task is completed, Insight can output and display the target retrieval file, provide the download address of the target retrieval file, and provide the option of generating a reverse sql file with one click. The reverse sql file may refer to the one that is backtracked and generated by analyzing the input point of the database. The download address of the target retrieval file is obtained by taking the file generation path of the subtask in the table record based on the main task ID. The main task ID can be understood as GTMGtid.

[0056] Embodiment 2

[0057] Figure 2 This is a flowchart of a distributed database change data retrieval method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the target retrieval result. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0058] S210: Obtain search configuration information corresponding to the target search task.

[0059] S220: Determine a target data change log corresponding to the target retrieval task based on the full logical log of the distributed database and the first retrieval configuration information.

[0060] The retrieval configuration information includes: first retrieval configuration information and second retrieval configuration information. The first retrieval configuration information can be used to filter the logical log before parsing. The second retrieval configuration information can be used to filter the log parsing information after parsing. The target data change log can refer to the logical log that meets the first retrieval configuration information.

[0061] Specifically, the first search configuration information is used to screen the full logical log of the distributed database to determine the target data change log corresponding to the target search task.

[0062] S230: Analyze the target data change log to determine the target analysis information corresponding to the target retrieval task.

[0063] The database data change log records the data change operations performed on the business database. By parsing the database data change log, the execution time of the database change event and the corresponding operation statement and other related information, namely the target parsing information, can be obtained.

[0064] Specifically, the DBAgent parses the target data change log to determine the target parsing information corresponding to the target retrieval task.

[0065] S240: Determine a target search result corresponding to the target search task based on the second search configuration information and the target resolution information.

[0066] Specifically, the second search configuration information is used to screen the parsed logs, that is, the target parsing information, to determine the target search result corresponding to the target search task.

[0067] S250: Determine a target retrieval file corresponding to the target retrieval task based on a preset output format, a current task state corresponding to the target retrieval task, and a target retrieval result.

[0068] The technical solution of the embodiment of the present invention determines the target data change log corresponding to the target retrieval task based on the full logical log of the distributed database and the first retrieval configuration information, thereby realizing preliminary retrieval before analysis; parses the target data change log to determine the target analysis information corresponding to the target retrieval task; determines the target retrieval result corresponding to the target retrieval task based on the second retrieval configuration information and the target analysis information, thereby realizing multi-dimensional detailed retrieval after analysis, and further improving the efficiency and accuracy of change data retrieval through multi-dimensional retrieval.

[0069] On the basis of the above technical solution, the retrieval configuration information is table dimension retrieval information; wherein, the table dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier, a library table name, a specified column filtering condition, an action type, and a memory limit during the retrieval process; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier, a library table name, a specified column filtering condition, an action type, and a memory limit during the retrieval process.

[0070] Among them, table dimension retrieval information can refer to the filtering information for each data in the table. Parsing logical logs requires space, so the time range needs to be used to control the parsing range. The addition, deletion, and modification operations of the host, backup machine, and specified node will be synchronized to the logical logs of all backup machines. According to the sharding strategy, a backup machine in the shard can be selected for parsing query, so a subtask corresponding to the shard is also generated. The shard identifier can be all shard identifiers or specified shard identifiers. All shards are tenant-level granularity. Field storage is based on the table granularity, so it is necessary to specify the library table name to narrow the scope to avoid duplicate columns in multiple tables and search out too much data.

[0071] The specified column filter condition can include the new value and old value of the specified column. For example, col[i]: the column value of the i-th column, i starts from the minimum value of 1, such as: value[1] represents the column name of column 1; col[i].old: the old value of the i-th column; col[i].new: the new value of the i-th column. For example: (col[1].old==5and col[1].new==6)and(col

[20] .old==5and col

[20] .new==6) The result is the operation statement that the column 1 and column 20 of the specified library table are changed from the old value 5 to the new value 6.

[0072] Action types may include: insert, update, and delete. Action types support multiple selections. Memory limits during the search process may refer to controlling the size of the search result file, such as the default of 100M. If it exceeds 100M, the task is interrupted. The embodiment of the present invention supports matching and screening of multi-dimensional filtering conditions to obtain target search results.

[0073] Based on the above technical solution, the retrieval configuration information is transaction dimension retrieval information; wherein the transaction dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier and a transaction size; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier and a transaction size.

[0074] Among them, the transaction dimension retrieval information may refer to the screening information for each data of the transaction. In the embodiment of the present invention, the transaction may refer to the data change operation. Some SQL statements executed inside the transaction involve a large number of data rows and a large amount of data, so the transaction dimension retrieval can be used to search for the transaction size. The transaction size can be used to limit the transaction size. For example, to find a large transaction, enter the transaction size of 50M, and you can find transactions larger than 50M.

[0075] The embodiment of the present invention supports retrieval at the library table level and transaction size dimension, and the retrieval of the table dimension can also specify column filtering conditions, specify the new value and old value of the column, and can also find out how a row of data is updated with one click.

[0076] Exemplarily, the parameter parsing thread in the DBAgent is used to check the validity of each parameter in the retrieved retrieval configuration information. When all parameters in the retrieval configuration information are valid, the control thread in the DBAgent is used to filter out a valid logical log analysis file list according to the time range. The logical log analysis file list is used to store the target data change log. If the logical log analysis file list is empty, the target retrieval task is terminated. If the logical log analysis file list is not empty, the analysis thread and the output thread are started. The analysis thread in the DBAgent is used to parse the target data change log, determine the target parsing information corresponding to the target retrieval task, and determine the target retrieval result corresponding to the target retrieval task based on the second retrieval configuration information and the target parsing information. The specific process of determining the target retrieval result includes: screening layer by layer according to each screening condition in the second retrieval configuration information, and if the screening result of a certain layer is empty, the target retrieval task is terminated. Otherwise, the screening is continued to obtain the target retrieval result. The output thread in the DBAgent is used to output the result file according to the specified output row format, that is, to determine the target retrieval file based on the preset output format and the target retrieval result.

[0077] Among them, in the process of layer-by-layer screening, if the column expression is not empty, the expression operation is implemented through the Lua script to obtain the operation result; if the column expression is empty, the expression operation is skipped. The forward SQL statement is generated based on the row event (RowEvent) structure. The row event structure in the database can refer to the structure used to record data change events during database replication or change data capture (CDC). These event structures describe how data is modified, including insert, update, and delete operations.

[0078] The following is an embodiment of a distributed database change data retrieval device provided in an embodiment of the present invention. The device and the distributed database change data retrieval method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the distributed database change data retrieval device, please refer to the embodiment of the above-mentioned distributed database change data retrieval method.

[0079] Embodiment 3

[0080] Figure 3 This is a schematic diagram of the structure of a distributed database change data retrieval device provided in Embodiment 3 of the present invention. Figure 3As shown, the device includes: a retrieval configuration information acquisition module 310, a target retrieval result determination module 320 and a target retrieval file determination module 330.

[0081] Among them, the retrieval configuration information acquisition module 310 is used to obtain the retrieval configuration information corresponding to the target retrieval task; the target retrieval result determination module 320 is used to determine the target retrieval result corresponding to the target retrieval task based on the full logical log and retrieval configuration information of the distributed database; the target retrieval file determination module 330 is used to determine the target retrieval file corresponding to the target retrieval task based on the preset output format, the current task status corresponding to the target retrieval task and the target retrieval result.

[0082] The technical solution of the embodiment of the present invention obtains the retrieval configuration information corresponding to the target retrieval task; determines the target retrieval result corresponding to the target retrieval task based on the full logical log and retrieval configuration information of the distributed database; determines the target retrieval file corresponding to the target retrieval task based on a preset output format, the current task status corresponding to the target retrieval task and the target retrieval result, thereby realizing one-click retrieval of changed data without human intervention, thereby improving the efficiency and accuracy of changed data retrieval.

[0083] On the basis of the above technical solution, the retrieval configuration information includes: first retrieval configuration information and second retrieval configuration information;

[0084] The target retrieval result determination module 320 is specifically used to: determine the target data change log corresponding to the target retrieval task based on the full logical log of the distributed database and the first retrieval configuration information; parse the target data change log to determine the target resolution information corresponding to the target retrieval task; and determine the target retrieval result corresponding to the target retrieval task based on the second retrieval configuration information and the target resolution information.

[0085] On the basis of the above technical solution, the retrieval configuration information is table dimension retrieval information; wherein, the table dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier, a library table name, a specified column filtering condition, an action type, and a memory limit during the retrieval process; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier, a library table name, a specified column filtering condition, an action type, and a memory limit during the retrieval process.

[0086] Based on the above technical solution, the retrieval configuration information is transaction dimension retrieval information; wherein the transaction dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier and a transaction size; the first retrieval configuration information includes a time range; the second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier and a transaction size.

[0087] Based on the above technical solution, the target retrieval file determination module 330 is specifically used to: if the current task status corresponding to the target retrieval task is the completed status, then based on the preset output format and the target retrieval result corresponding to the target retrieval task, determine the target retrieval file corresponding to the target retrieval task.

[0088] On the basis of the above technical solution, the device also includes:

[0089] A current task state determination module is used to detect the current subtask state corresponding to each subtask in the distributed database, and determine the current task state corresponding to the target retrieval task based on the current subtask state corresponding to each subtask;

[0090] Among them, the subtask corresponds to the shard identifier.

[0091] The distributed database change data retrieval device provided in the embodiment of the present invention can execute the distributed database change data retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the distributed database change data retrieval method.

[0092] It is worth noting that in the above-mentioned embodiment of distributed database change data retrieval, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0093] Embodiment 4

[0094] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0095] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0096] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a distributed database change data retrieval method.

[0098] In some embodiments, the distributed database change data retrieval method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the distributed database change data retrieval method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the distributed database change data retrieval method in any other appropriate manner (e.g., by means of firmware).

[0099] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0101] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0104] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0105] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the distributed database change data retrieval method provided in any embodiment of the present application.

[0106] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The program product and the distributed database change data retrieval method disclosed in each embodiment of the present application belong to the same inventive concept, so it is not repeated here.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0108] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed database change data retrieval method, characterized in that: include: Obtain the retrieval configuration information corresponding to the target retrieval task; Determine the target search result corresponding to the target search task based on the full logical log of the distributed database and the search configuration information; Based on a preset output format, a current task state corresponding to the target retrieval task and the target retrieval result, a target retrieval file corresponding to the target retrieval task is determined.

2. The method according to claim 1, characterized in that The retrieval configuration information includes: first retrieval configuration information and second retrieval configuration information; The determining of the target search result corresponding to the target search task based on the full logical log of the distributed database and the search configuration information includes: Determine a target data change log corresponding to the target retrieval task based on the full logical log of the distributed database and the first retrieval configuration information; Parsing the target data change log to determine the target parsing information corresponding to the target retrieval task; Based on the second search configuration information and the target resolution information, a target search result corresponding to the target search task is determined.

3. The method according to claim 2, characterized in that The retrieval configuration information is table dimension retrieval information; The table dimension retrieval information includes: at least one of: time range, sharding strategy, sharding identifier, library table name, specified column filtering condition, action type, and memory limit during retrieval; The first search configuration information includes a time range; The second search configuration information includes: at least one of a sharding strategy, a sharding identifier, a library table name, a specified column filtering condition, an action type, and a memory limit during the search process.

4. The method according to claim 2, characterized in that: The retrieval configuration information is transaction dimension retrieval information; The transaction dimension retrieval information includes: at least one of a time range, a sharding strategy, a sharding identifier, and a transaction size; The first search configuration information includes a time range; The second retrieval configuration information includes: at least one of a sharding strategy, a sharding identifier, and a transaction size.

5. The method according to claim 1, characterized in that: The determining the target retrieval file corresponding to the target retrieval task based on the preset output format, the current task state corresponding to the target retrieval task and the target retrieval result includes: If the current task status corresponding to the target retrieval task is a completed status, the target retrieval file corresponding to the target retrieval task is determined based on a preset output format and the target retrieval result corresponding to the target retrieval task.

6. The method according to claim 5, characterized in that The method further comprises: Detecting the current subtask state corresponding to each subtask in the distributed database, and determining the current task state corresponding to the target retrieval task based on the current subtask state corresponding to each subtask; The subtask corresponds to the slice identifier.

7. A distributed database change data retrieval device, characterized in that: The device comprises: A retrieval configuration information acquisition module is used to obtain the retrieval configuration information corresponding to the target retrieval task; A target search result determination module, used to determine the target search result corresponding to the target search task based on the full logical log of the distributed database and the search configuration information; The target retrieval file determination module is used to determine the target retrieval file corresponding to the target retrieval task based on a preset output format, a current task state corresponding to the target retrieval task and the target retrieval result.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed database change data retrieval method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed database change data retrieval method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the distributed database change data retrieval method as described in any one of claims 1 to 6.

Images