Distributed database data change retrieval method and device, equipment and medium

Abstract

The invention discloses a distributed database data change retrieval method and device, equipment and a medium, through a specific data change retrieval strategy, a binary log file needing to be retrieved can be quickly positioned, comprehensive scanning of a whole database log is avoided, and therefore the retrieval efficiency is improved; according to the method, a user can customize a retrieval strategy according to needs, accurate matching is carried out by utilizing a time range, a table name, a column filtering condition and an operation type, it is ensured that only change records meeting specific conditions are retrieved and analyzed, processing of irrelevant data is reduced, and retrieval accuracy is improved; meanwhile, data change retrieval in a distributed database environment is supported, data can be integrated and analyzed in a cross-node mode, and the technical problems that due to the distributed storage characteristic of data, change records are possibly distributed at different physical positions, data integration and analysis need to be conducted in a cross-node mode, and the searching efficiency is low are solved.

Description

Technical Field

[0001] The present application relates to the field of database management technology, and in particular to a distributed database data change retrieval method, device, equipment and medium. Background Art

[0002] In a rapidly changing business environment, continuous updating and maintenance of data is the norm. For frequently updated data records, application systems often need to track and count these changes in order to facilitate in-depth analysis and problem location. For example, in the fields of financial services, online transactions, and real-time monitoring systems, monitoring and analysis of data changes are particularly important. These changes may involve key business data such as updating account information, adding or removing transaction records, and recording user behavior.

[0003] However, current practices often rely on manual parsing of binary log files, which has many limitations. First, the parsing of binary files requires professional skills and a lot of time, which is a high threshold for non-professionals. Second, as the amount of data continues to grow, the size of log files is also expanding, and the difficulty and workload of manual parsing are increasing. In addition, there may be omissions in the manual parsing process, resulting in incomplete or inaccurate records of data changes.

[0004] In a distributed database system, data is stored in multiple nodes or shards, which further increases the complexity of tracking data changes. Due to the distributed storage characteristics of data, change records may be distributed in different physical locations, requiring cross-node data integration and analysis. With a large number of shards, this cross-node data integration becomes more difficult, not only needing to deal with network latency and data consistency issues, but also solving the challenges of data merging and synchronization.

[0005] Therefore, the database needs to have the function of one-click retrieval of data change actions to avoid data center operation and maintenance personnel spending a lot of energy to parse the database binary logs. Summary of the invention

[0006] The present application provides a distributed database data change retrieval method, device, equipment and medium, which solves the technical problem of low search efficiency due to the distributed storage characteristics of data, where change records may be distributed in different physical locations and data integration and analysis need to be performed across nodes.

[0007] In view of this, the first aspect of the present application provides a distributed database data change retrieval method, the method comprising:

[0008] Get data change retrieval strategy;

[0009] Reading a binary log file of a target database according to the data change retrieval strategy, wherein the binary log file is used to record change operations on the target database;

[0010] Filter the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range;

[0011] Matching the binary analysis file list according to the input parameter matching table name, column filtering condition and operation type in the data change retrieval strategy;

[0012] If the column expression of the successfully matched binary analysis file is not empty, the preset Lua script is called to perform expression calculation and generate a forward SQL statement according to the RowEvent structure;

[0013] Transaction information and the forward SQL statement are distributed to the output thread, wherein the transaction information includes the successfully matched binary analysis file, and the output thread analyzes whether the binary analysis file satisfies the data change search strategy and generates a search result.

[0014] Optionally, the data change retrieval strategy specifically includes:

[0015] The user configures and generates a data change retrieval strategy through the front-end page. The data change retrieval strategy includes time range, sharding strategy, sharding location, input parameter matching table name, column filtering condition, operation type, and memory control information;

[0016] The data change retrieval strategy is sent to the message management module through the front-end page.

[0017] Optionally, reading the binary log file of the target database according to the data change retrieval strategy specifically includes:

[0018] The message management module determines the target database according to the sharding strategy and the sharding location in the data change retrieval strategy, and sends the data change retrieval strategy to the execution retrieval program module corresponding to the target database;

[0019] The execution retrieval program module reads the binary log file of the target database.

[0020] Optionally, after the executing retrieval program module reads the binary log file of the target database, the method further includes:

[0021] The execution retrieval program module performs a validity check on the binary log file by performing parameter parsing on the binary log file in the memory;

[0022] If valid, initialize the thread resources, otherwise end the retrieval process.

[0023] Optionally, filtering the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range further includes:

[0024] Start several analysis threads and output threads;

[0025] According to the memory control information in the data change retrieval strategy, several binary analysis files included in the binary analysis file list are distributed as transaction information to each of the analysis threads for concurrent execution, and the concurrency number of the analysis threads is determined by the memory control information.

[0026] Optionally, distributing the plurality of binary analysis files included in the binary analysis file list as transaction information to each of the analysis threads for concurrent execution further includes:

[0027] If the transaction information of the binary analysis file exceeds the transaction threshold corresponding to the memory control information, split the transaction information into a plurality of partial transaction information;

[0028] Distribute the plurality of partial transaction information to the same analysis thread in sequence.

[0029] Optionally, after generating the search results, the step further includes:

[0030] Regularly detecting the search result generated by the execution search program module;

[0031] The search results are returned to the front-end page, and the front-end page displays the search result information.

[0032] A second aspect of the present application provides a distributed database data change retrieval device, the device comprising:

[0033] An acquisition unit, used for acquiring a data change retrieval strategy;

[0034] A reading unit, used to read a binary log file of a target database according to the data change retrieval strategy, wherein the binary log file is used to record the change operation on the target database;

[0035] A filtering unit, configured to filter the binary log files according to a time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range;

[0036] A matching unit, used to match the binary analysis file list according to the input parameter matching table name, column filtering condition and operation type in the data change retrieval strategy;

[0037] The processing unit is used to call the preset Lua script to perform expression calculation if the column expression of the successfully matched binary analysis file is not empty, and generate a forward SQL statement according to the RowEvent structure;

[0038] The output unit is used to distribute transaction information and the forward SQL statement to the output thread, wherein the transaction information includes the successfully matched binary analysis file, and the output thread analyzes whether the binary analysis file satisfies the data change retrieval strategy and generates a retrieval result.

[0039] A third aspect of the present application provides a distributed database data change retrieval device, the device comprising a processor and a memory:

[0040] The memory is used to store program code and transmit the program code to the processor;

[0041] The processor is used to execute the steps of the distributed database data change retrieval method as described in the first aspect according to the instructions in the program code.

[0042] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0043] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0044] In the present application, a distributed database data change retrieval method, device, equipment and medium are provided. Through a specific data change retrieval strategy, the binary log file to be retrieved can be quickly located, avoiding a comprehensive scan of the entire database log, thereby improving the retrieval efficiency; the user can customize the retrieval strategy as needed, and use the time range, table name, column filter condition and operation type for precise matching, ensuring that only change records that meet specific conditions are retrieved and analyzed, reducing the processing of irrelevant data and improving the accuracy of the retrieval; at the same time, it supports data change retrieval in a distributed database environment, and can integrate and analyze data across nodes, solving the technical problem of low search efficiency due to the distributed storage characteristics of the data, where change records may be distributed in different physical locations and data integration and analysis need to be performed across nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1A method flow chart of a distributed database data change retrieval method in an embodiment of the present application;

[0046] Figure 2 This is a schematic diagram of the structure of a distributed database data change retrieval device in an embodiment of the present application;

[0047] Figure 3 It is a structural diagram of a distributed database data change retrieval device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The present application designs a distributed database data change retrieval method, device, equipment and medium, which solves the technical problem of low search efficiency due to the distributed storage characteristics of data, where change records may be distributed in different physical locations and data integration and analysis need to be performed across nodes.

[0050] For easier understanding, see Figure 1 , Figure 1 Flowchart of the distributed database data change retrieval method in the embodiment of the present application. Figure 1 As shown, specifically:

[0051] Step S1, obtaining data change retrieval strategy;

[0052] It should be noted that the user inputs or selects parameters such as time range, sharding strategy, sharding location, table name, column filter condition, operation type and memory control through the front-end page to generate a complete data change retrieval strategy.

[0053] For example, the user sets the time range to January 1, 2024 to January 31, 2024, the table name is orders, and the operation type is Insert.

[0054] Step S2: Read the binary log file of the target database according to the data change retrieval strategy, where the binary log file is used to record the change operation on the target database;

[0055] It should be noted that the execution retrieval program module reads the binary log file of the target database according to the received strategy. The binary log file is used to record all change operations on the database, but does not include query operations. These change operations include modifications to data, such as inserting, updating, and deleting, but do not include query operations on database tables and contents.

[0056] Step S3: Filter the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range;

[0057] It should be noted that the valid binary analysis file list is filtered out according to the time range. If it is empty, the process will exit, otherwise it will continue.

[0058] Step S4, matching the binary analysis file list according to the input parameter matching table name, column filter condition and operation type in the data change retrieval strategy;

[0059] It should be noted that the analysis thread matches the transaction information according to the table name, column filter condition and operation type in the retrieval strategy.

[0060] Among them, the operation types are Insert, Update, and Delete, which are consistent with the SQL action types configured on the front-end page.

[0061] Step S5: If the column expression of the successfully matched binary analysis file is not empty, call the preset Lua script to perform expression calculation and generate a forward SQL statement according to the RowEvent structure;

[0062] It should be noted that if the match is successful and the column expression is not empty, the Lua script is called to perform expression calculation and generate a forward SQL statement.

[0063] For example:

[0064] The expression col[2].new == 'abcd' retrieves the SQL for inserting or updating the new value of the second field which is abcd;

[0065] col[2].old == 'abcd' retrieves the SQL where the second field of the old value of delele or update is abcd;

[0066] (col[2].new=='abcd')and(col[3].old=='abcd') retrieves the update sql where the new value of the second field is abcd and the old value of the third field is abcd.

[0067] If yes, continue; otherwise, skip calling the Lua script to perform expression calculation.

[0068] Generate forward SQL statements based on RowEvent and other structures to wake up the output thread.

[0069] Step S6: Distribute transaction information and forward SQL statements to the output thread. The transaction information includes a successfully matched binary analysis file. The output thread analyzes whether the binary analysis file meets the data change search strategy and generates a search result.

[0070] It should be noted that if there is a redo dql statement, the transaction information and the redo sql statement will be distributed to the output thread at the same time; if the memory control conditions are met, there will be corresponding memory for the output thread, and the log content will be analyzed to see if it meets the retrieval conditions.

[0071] The output line format of the SQL file corresponding to the search results finally generated can be:

[0072] Transaction number, transaction size, time, file location, binary file location, SQL statement.

[0073] Furthermore, the data change retrieval strategy specifically includes:

[0074] Users configure and generate data change retrieval policies through the front-end page. Data change retrieval policies include time range, sharding strategy, sharding location, input parameter matching table name, column filtering conditions, operation type, and memory control information.

[0075] Send data change retrieval strategy to the message management module through the front-end page.

[0076] It should be noted that the user inputs or selects parameters such as time range, sharding strategy, sharding location, table name, column filter condition, operation type and memory control through the front-end page to generate a complete data change retrieval strategy.

[0077] For example, the user sets the time range to January 1, 2024 to January 31, 2024, the table name is orders, and the operation type is Insert.

[0078] After the user submits the search strategy on the front-end page, the strategy is sent to the back-end through the message management module.

[0079] Furthermore, reading the binary log file of the target database according to the data change retrieval strategy specifically includes:

[0080] The message management module determines the target database according to the sharding strategy and sharding location in the data change retrieval strategy, and sends the data change retrieval strategy to the execution retrieval program module corresponding to the target database;

[0081] The retrieval program module is executed to read the binary log file of the target database.

[0082] It should be noted that the message management module determines the target database according to the sharding strategy and location, and sends the search strategy to the execution search program module.

[0083] The execution retrieval program module reads the binary log file of the target database according to the received strategy.

[0084] Further, after executing the retrieval program module to read the binary log file of the target database, the following steps are also included:

[0085] The execution retrieval program module performs a validity check on the binary log file by parsing the parameters of the binary log file in the memory;

[0086] If valid, initialize the thread resources, otherwise end the retrieval process.

[0087] It should be noted that the execution retrieval program module parses the binary log file in the memory to check whether the log file is complete and not damaged.

[0088] If the log file is invalid, the retrieval process ends; if valid, proceed to the next step.

[0089] Among them, executing the retrieval program to parse the parameters of the binary log file in the memory will not generate temporary files to occupy disk space and no need to clean up.

[0090] Furthermore, after filtering the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range, the following is also included:

[0091] Start several analysis threads and output threads;

[0092] According to the memory control information in the data change retrieval strategy, several binary analysis files included in the binary analysis file list are distributed as transaction information to each analysis thread for concurrent execution, and the concurrent number of analysis threads is determined by the memory control information.

[0093] It should be noted that the files in the binary analysis file list are distributed to each analysis thread according to the memory control information.

[0094] For example, if the memory control information allows five files to be processed simultaneously, the system will distribute the five files to five analysis threads.

[0095] Furthermore, distributing the plurality of binary analysis files included in the binary analysis file list as transaction information to each analysis thread for concurrent execution also includes:

[0096] If the transaction information of the binary analysis file exceeds the transaction threshold corresponding to the memory control information, the transaction information is split into several partial transaction information;

[0097] Distribute several partial transaction information to the same analysis thread in sequence.

[0098] It should be noted that if a single transaction information exceeds the memory threshold, it will be split into multiple partial transaction information to avoid memory overflow.

[0099] For example, if a large transaction contains 100 mutation operations and exceeds the memory limit, it can be split into 20 parts, each with 5 operations.

[0100] Furthermore, after generating the search results, the following steps are also included:

[0101] Regularly detect and execute the search results generated by the search program module;

[0102] The search results are returned to the front-end page, which displays the search result information.

[0103] It should be noted that the execution of the search program is an asynchronous thread. The database program proxy module will detect the return results of the execution of the search program at regular intervals, organize and output the result files and return them to the front-end page. The front-end page provides detailed information about the search task and the search status information of each shard during the search process. After the search task is completed, the front-end page provides a download function.

[0104] See also Figure 2 , Figure 2 Schematic diagram of the structure of the distributed database data change retrieval device in the embodiment of the present application. Figure 2 As shown, specifically:

[0105] An acquisition unit 201 is used to acquire a data change retrieval strategy;

[0106] A reading unit 202 is used to read a binary log file of a target database according to a data change retrieval strategy, wherein the binary log file is used to record change operations on the target database;

[0107] A filtering unit 203 is used to filter the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range;

[0108] A matching unit 204, used to match the binary analysis file list according to the input parameter matching table name, column filtering condition and operation type in the data change search strategy;

[0109] Processing unit 205, for calling a preset Lua script to perform expression calculation if the column expression of the successfully matched binary analysis file is not empty, and generating a forward SQL statement according to the RowEvent structure;

[0110] The output unit 206 is used to distribute transaction information and forward SQL statements to the output thread. The transaction information includes a successfully matched binary analysis file. The output thread analyzes whether the binary analysis file meets the data change search strategy and generates a search result.

[0111] The present application also provides another distributed database data change retrieval device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0112] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0113] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0114] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0115] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0116] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0117] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0118] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0119] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0120] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0121] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0122] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0123] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0124] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0125] Get data change retrieval strategy;

[0126] Read the binary log file of the target database according to the data change retrieval strategy. The binary log file is used to record the change operations on the target database.

[0127] Filter the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range;

[0128] Match the binary analysis file list according to the input parameter matching table name, column filter condition, and operation type in the data change retrieval strategy;

[0129] If the column expression of the successfully matched binary analysis file is not empty, the preset Lua script is called to perform expression calculation and generate a forward SQL statement according to the RowEvent structure;

[0130] Distribute transaction information and forward SQL statements to the output thread. The transaction information contains the successfully matched binary analysis file. The output thread analyzes whether the binary analysis file meets the data change retrieval strategy and generates the retrieval result.

[0131] The embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of a distributed database data change retrieval method described in the aforementioned embodiments.

[0132] In the embodiments of the present application, a distributed database data change retrieval method, apparatus, device and medium are provided. Through a specific data change retrieval strategy, the binary log file to be retrieved can be quickly located, avoiding a comprehensive scan of the entire database log, thereby improving the retrieval efficiency; the user can customize the retrieval strategy as needed, and use the time range, table name, column filter condition and operation type for precise matching, ensuring that only change records that meet specific conditions are retrieved and analyzed, reducing the processing of irrelevant data and improving the accuracy of the retrieval; at the same time, it supports data change retrieval in a distributed database environment, and can integrate and analyze data across nodes, solving the technical problem of low search efficiency due to the distributed storage characteristics of the data, where change records may be distributed in different physical locations and data integration and analysis need to be performed across nodes.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0135] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0140] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed database data change retrieval method, characterized in that: include: Get data change retrieval strategy; Reading a binary log file of a target database according to the data change retrieval strategy, wherein the binary log file is used to record change operations on the target database; Filter the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range; Matching the binary analysis file list according to the input parameter matching table name, column filtering condition and operation type in the data change retrieval strategy; If the column expression of the successfully matched binary analysis file is not empty, the preset Lua script is called to perform expression calculation and generate a forward SQL statement according to the RowEvent structure; Transaction information and the forward SQL statement are distributed to the output thread, wherein the transaction information includes the successfully matched binary analysis file, and the output thread analyzes whether the binary analysis file satisfies the data change search strategy and generates a search result.

2. The distributed database data change retrieval method according to claim 1, characterized in that: The acquisition data change retrieval strategy specifically includes: The user configures and generates a data change retrieval strategy through the front-end page. The data change retrieval strategy includes time range, sharding strategy, sharding location, input parameter matching table name, column filtering condition, operation type, and memory control information; The data change retrieval strategy is sent to the message management module through the front-end page.

3. The distributed database data change retrieval method according to claim 2, characterized in that: The step of reading the binary log file of the target database according to the data change retrieval strategy specifically includes: The message management module determines the target database according to the sharding strategy and the sharding location in the data change retrieval strategy, and sends the data change retrieval strategy to the execution retrieval program module corresponding to the target database; The execution retrieval program module reads the binary log file of the target database.

4. The distributed database data change retrieval method according to claim 3, characterized in that: After the execution retrieval program module reads the binary log file of the target database, the method further includes: The execution retrieval program module performs a validity check on the binary log file by performing parameter parsing on the binary log file in the memory; If valid, initialize the thread resources, otherwise end the retrieval process.

5. The distributed database data change retrieval method according to claim 4, characterized in that: After filtering the binary log files according to the time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range, the following further comprises: Start several analysis threads and output threads; According to the memory control information in the data change retrieval strategy, several binary analysis files included in the binary analysis file list are distributed as transaction information to each of the analysis threads for concurrent execution, and the concurrency number of the analysis threads is determined by the memory control information.

6. The distributed database data change retrieval method according to claim 5, characterized in that: The distributing the plurality of binary analysis files included in the binary analysis file list as transaction information to each of the analysis threads for concurrent execution also includes: If the transaction information of the binary analysis file exceeds the transaction threshold corresponding to the memory control information, split the transaction information into a plurality of partial transaction information; Distribute the plurality of partial transaction information to the same analysis thread in sequence.

7. The distributed database data change retrieval method according to claim 6, characterized in that: After generating the search results, the following steps are further included: Regularly detecting the search result generated by the execution search program module; The search results are returned to the front-end page, and the front-end page displays the search result information.

8. A distributed database data change retrieval device, characterized in that: include: An acquisition unit, used for acquiring a data change retrieval strategy; A reading unit, used to read a binary log file of a target database according to the data change retrieval strategy, wherein the binary log file is used to record the change operation on the target database; A filtering unit, configured to filter the binary log files according to a time range in the data change retrieval strategy to obtain a list of binary analysis files that meet the time range; A matching unit, used to match the binary analysis file list according to the input parameter matching table name, column filtering condition and operation type in the data change retrieval strategy; The processing unit is used to call the preset Lua script to perform expression calculation if the column expression of the successfully matched binary analysis file is not empty, and generate a forward SQL statement according to the RowEvent structure; The output unit is used to distribute transaction information and the forward SQL statement to the output thread, wherein the transaction information includes the successfully matched binary analysis file, and the output thread analyzes whether the binary analysis file satisfies the data change retrieval strategy and generates a retrieval result.

9. A distributed database data change retrieval device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the distributed database data change retrieval method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the distributed database data change retrieval method according to any one of claims 1 to 7.

Images