A method, device and medium for capturing system change data
By obtaining process information of business systems from non-relational unstructured databases, intercepting code injection and caching data, the accuracy problem of change data capture is solved, and unified capture of structured and semi-structured data is achieved. It has wide applicability and is easy to maintain.
Patent Information
- Application Number
- CN202411186596.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-28
AI Technical Summary
Existing technologies struggle to accurately capture changed data when dealing with non-relational, unstructured databases.
By acquiring process information from the business system, determining the language type, and performing interception code injection, recording database operation commands, and using pre-built programming interfaces and data caching tasks to capture changed data, the system can capture changed data for both structured and semi-structured data.
Without altering the original code's business logic, it achieves change data capture for different types of databases, has wide applicability, and uses a unified method that is easy to maintain.
Smart Images

Figure CN118689894B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of electronic digital data processing, and in particular to a method, device and medium for capturing system change data. Background Art
[0002] The existing data in the database is generally large in volume, and the number of records involved in daily changes is relatively small. However, when performing tasks such as data synchronization, analysis, and calculation, it is usually necessary to obtain the full amount of data. CDC (Change Data Capture) is a method for capturing change data that can monitor and obtain data that is added, updated, and deleted in a system. Generally, CDC refers to the capture of change data for structured relational databases. It focuses on changes to table records on relational database tables, including operations such as adding records, updating records, and deleting records.
[0003] Secondly, the existing data capture methods based on table snapshot comparison, database trigger mechanism, and database log only capture change data for relational databases with structured data and databases that provide certain specific mechanisms. With the advent of the big data era, the demand for different types of data is gradually increasing, making it difficult to accurately obtain change data when facing non-relational and unstructured databases. Summary of the invention
[0004] The embodiments of the present application provide a system change data capture method, device and medium, which are used to solve the following technical problem: when facing a non-relational and unstructured database, it is difficult to accurately obtain the change data.
[0005] The present application embodiment adopts the following technical solutions:
[0006] The embodiment of the present application provides a method for capturing system change data. The method includes obtaining process information of a business system, determining a corresponding language type based on the process information, and injecting interception code into the business system based on the language type and a preset programming interface; wherein the injected interception code is used to record the input parameter value of the preset programming interface; after the database system receives a data change request, the changed data is recorded separately through a preset data cache task; before returning a response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured; and the captured change data is sent to the target end in the order of capture to complete the synchronization of the change data.
[0007] The embodiment of the present application intercepts code injection into the business system based on the language type and the preset programming interface, records the operation commands of the business system on the database without changing the original code business logic, and then monitors the commands or SQL statements executed by the business system on the database to complete the capture of the changed data. The embodiment of the present application can not only realize the capture of changed data of structured data, but also realize the capture of changed data of semi-structured data (such as KV type data), and has a wide range of applications. And for different structured and unstructured databases, the same technical means are used to realize the capture of changed data, and the method is unified and easy to maintain.
[0008] In one implementation of the present application, code injection is intercepted into a business system based on a language type and a preset programming interface, specifically including: based on the language type, determining the code to be injected, and compiling the code to be injected into a shared object file; attaching the code injection tool to the service process corresponding to the business system, and obtaining register information corresponding to the service process; based on the libc.so base address corresponding to the target process, the libc.so base address corresponding to the local process, and the required function address corresponding to the local process, determining the required function address corresponding to the service process; wherein the required function address includes at least one of the mmap address, the dlopen address, the dlsym address, and the dlclose address; calling the required function, and injecting the shared object file into the service process corresponding to the business system based on the required function; remotely calling the detach function to detach from the target process, so that the service process corresponding to the business system continues to run, so as to complete the injection of the intercepted code.
[0009] In one implementation of the present application, the required function is called, and based on the required function, the shared object file is injected into the service process corresponding to the business system, specifically including: calling the mmap function to save the parameter information by applying for memory space; writing the shared object file into the applied memory space, and remotely calling the dlopen function to open the written shared object file; calling the dlsym function to load the target function address corresponding to the service process, writing the target function parameters, remotely calling the target function and executing it; remotely calling the dlclose function to unload the shared object file and restore the original register environment.
[0010] In one implementation of the present application, after the database system receives a data change request, the changed data is recorded separately through a preset data cache task, specifically including: when any data change method of the database system is called, a preset background asynchronous task is started to record the changed data separately; or, when any data change method of the database system is called, the changed data is recorded separately through a preset cache item; wherein the preset cache item includes at least one item in a message queue and a memory database.
[0011] In one implementation of the present application, after the database system receives a data change request, the method also includes: when receiving a data change request initiation command, uniquely identifying the data corresponding to the data change command and caching the marked data; after receiving the data change request submission information, sending the marked data to the corresponding target end; after receiving the data change request rollback information, based on the identification information in the rollback information, performing data query in the cached data, and deleting the queried data.
[0012] In one implementation of the present application, after caching the marked data, the method also includes: determining the data type corresponding to the marked data, and determining a reference data set with the same data type in a historical database; determining the cache duration corresponding to the data in the reference data set, and using the maximum cache duration as the first reference duration; determining in the historical database the average cache duration corresponding to multiple data within a preset time period before receiving the marked data, and using the average cache duration as the second reference duration; weighting the first reference duration and the second reference duration to obtain a data expiration duration; marking the marked data with an expiration time based on the data expiration time, and deleting the marked data from the cache when the expiration time is reached.
[0013] In one implementation of the present application, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured, specifically including: obtaining the data change result recorded in the response object through the injected interception code; wherein the data change result includes a successful change and other values; analyzing the data change result, and when it is determined that the data change result is a successful change, capturing the separately recorded change data; wherein the change data includes at least text data and semi-structured data; when it is determined that the data change result is other values, ending the current data capture task.
[0014] In one implementation of the present application, the captured change data is sent to the target end in the order of capture to complete the synchronization of the change data, specifically including: assigning a timestamp to each captured change data, and performing a priority identification based on the type of the captured change data; caching the captured change data into multiple data queues, and sorting the multiple change data in each data queue based on the priority identification; determining the target timestamps corresponding to the multiple data queues, and determining the target queue and the reference queue according to the target timestamps; wherein the target timestamp is the timestamp of the latest data corresponding to the data queues; according to the sorting order, the change data in the target queue is sent to the target end, and during the sending process, the target timestamps corresponding to the target queue and the reference queue are compared; when the target timestamp corresponding to the reference queue is faster than the timestamp corresponding to the target queue, and the timestamp difference is greater than a preset duration, the reference queue is used as the new target queue to send the change data.
[0015] An embodiment of the present application provides a system change data capture device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain process information of a business system, determine a corresponding language type based on the process information, and inject interception code into the business system based on the language type and a preset programming interface; wherein the injected interception code is used to record the input parameter value of the preset programming interface; after the database system receives a data change request, the change data is recorded separately through a preset data cache task; before returning a response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured; and the captured change data is sent to the target end in the order of capture to complete the synchronization of the change data.
[0016] A non-volatile computer storage medium provided by an embodiment of the present application stores computer executable instructions, wherein the computer executable instructions are configured to: obtain process information of a business system, determine a corresponding language type based on the process information, and inject interception code into the business system based on the language type and a preset programming interface; wherein the injected interception code is used to record input parameter values of the preset programming interface; after a database system receives a data change request, the changed data is recorded separately through a preset data cache task; before a response object generated after the database system is changed is returned to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured; and the captured change data is sent to a target end in the order of capture to complete the synchronization of the change data.
[0017] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the embodiments of the present application intercept code injection into the business system based on the language type and the preset programming interface, and record the operation commands of the business system on the database without changing the original code business logic, and then monitor the commands or SQL statements executed by the business system on the database to complete the capture of the changed data. The embodiments of the present application can not only realize the capture of changed data of structured data, but also realize the capture of changed data of semi-structured data (such as KV type data), and have a wide range of applications. And for different structured and unstructured databases, the same technical means are used to realize the capture of changed data, and the method is unified and easy to maintain. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:
[0019] Figure 1 A flow chart of a method for capturing system change data provided by an embodiment of the present application;
[0020] Figure 2 A schematic diagram showing the overall architecture comparison before and after a data interception code change provided in an embodiment of the present application;
[0021] Figure 3 A schematic diagram of a change data capture operation process provided in an embodiment of the present application;
[0022] Figure 4A summary diagram of change data capture provided by an embodiment of the present application;
[0023] Figure 5 A schematic diagram of the structure of a system change data capture device provided in an embodiment of the present application.
[0024] Reference numerals:
[0025] 200: system change data capture device, 201: processor, 202: memory. DETAILED DESCRIPTION
[0026] Embodiments of the present application provide a method, device, and medium for capturing system change data.
[0027] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0028] The technical solution proposed in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.
[0029] Figure 1 A flow chart of a method for capturing system change data provided by an embodiment of the present application is as follows: Figure 1 As shown, the system change data capture method includes the following steps:
[0030] Step 101: Obtain process information of the business system, determine the corresponding language type based on the process information, and intercept code injection into the business system based on the language type and a preset programming interface.
[0031] In one embodiment of the present application, the process information of all business systems is collected, and the computer programs that read and write DBMS (Database Management System) and the server information where they run are collected. For example, in the existing environment, there are computer programs A, B, and C that perform data reading and writing operations on DBMS, and the servers where they are located are a, b, and c respectively. Computer programs A, B, and C are collectively referred to as business systems. Among them, DBMS is a specific database system that needs to collect change data based on environmental assumptions.
[0032] Furthermore, collect and confirm the computer programming language used by each business system, which is generally C / C++, Java, Python, GoLang, etc. These programming languages all have a unified programming interface for DBMS operations. According to the programming language, clarify the specific implementation class and method name in the code library of the language's DBMS operation interface, as well as the method's function, entry parameters, return value and other information.
[0033] Furthermore, for each business system implemented in a programming language, code generation, code injection or code enhancement technology is used to inject and modify code in the implementation of the DBMS operation interface of the programming language standard. The method in the embodiment of the present application is to automatically adjust the program code before the program is loaded into the computer memory and starts to execute, so that the input parameter value of the interface can be recorded without changing the external function of the interface, that is, the injected interception code is used to record the input parameter value of the preset programming interface. At the method entrance of the DBMS operation code, the input parameter of the method, i.e., the DBMS operation statement, is recorded, thereby completing the interception and recording of the DBMS operation.
[0034] In one embodiment of the present application, based on the language type, the code to be injected is determined, and the code to be injected is compiled into a shared object file. The code injection tool is attached to the service process corresponding to the business system, and the register information corresponding to the service process is obtained. Based on the libc.so base address corresponding to the target process, the libc.so base address corresponding to the local process, and the required function address corresponding to the local process, the required function address corresponding to the service process is determined; wherein the required function address includes at least one of the mmap address, the dlopen address, the dlsym address, and the dlclose address. The required function is called, and based on the required function, the shared object file is injected into the service process corresponding to the business system. The detach function is remotely called to detach from the target process, so that the service process corresponding to the business system continues to run to complete the injection of the interception code.
[0035] In one embodiment of the present application, the mmap function is called to save the parameter information by applying for memory space. The shared object file is written into the applied memory space, and the dlopen function is remotely called to open the written shared object file. The dlsym function is called to load the target function address corresponding to the service process, write the target function parameters, remotely call the target function and execute it. The dlclose function is remotely called to unload the shared object file and restore the original register environment.
[0036] Specifically, different code injection tools can be used in different environments and programming languages. For example, under the Linux operating system, a business system implemented in C language can use tools such as Ptrace to implement code injection.
[0037] Specifically, the written interception code is injected into the business system process. First, the code to be injected is written and compiled into a so file, i.e. a shared object file. Using a code injection tool, this application embodiment takes Ptrace as an example and attaches it to the target process. The target process in the application embodiment is the service process corresponding to the business system. Save the register environment, obtain the current register status of the target process through the PTRACE_GETREGS parameter of ptrace, copy and save it, and facilitate the subsequent recovery of the register content; the code execution address is stored in the register.
[0038] Further, the base addresses of libc.so in the target process and the local process are obtained respectively, and then the mmap address, dlopen address, dlsym address and dlclose address in the local process are obtained, and the mmap address, dlopen address, dlsym address and dlclose address of the target program are determined by the formula of local function address - local libc.so base address + target process libc.so base address = function address in the target process. The mmap function is remotely called to apply for memory space to save parameter information, and the parameters of the mmap function are filled into the register through the PTRACE_POKETEXT and PTRACE_PEEKTEXT parameters of the ptrace function, and the register is set through the PTRACE_SETREGS parameter, and the child process is made to send a signal after continuing to execute the mmap function through the PTRACE_CONT parameter, and the on-site register content is saved again.
[0039] Furthermore, the return address of the mmap function is obtained, the so file to be injected is written to the address, and the dlopen function is remotely called to open the written so file. The dlsym function is remotely called to load the target function address, return the target function address, write the target function parameters, remotely call the target function and execute it, and remotely call the dlclose function to unload the so file.
[0040] Furthermore, the scene is restored, that is, the original environment of the register is restored, that is, the address of the next line of code to be executed stored in the register is restored, so that the program can start from the last execution position and continue to execute, and remotely call detach to detach from the target process so that the target process can continue to run. The code to be modified can be written into the target process, and then the code injection and modification are completed. The code resident in the computer memory of the business system has been changed.
[0041] Step 102: After the database system receives the data change request, the changed data is recorded separately by presetting the data cache task.
[0042] In one embodiment of the present application, when any data change method of the database system is called, a preset background asynchronous task is started to record the changed data separately. Alternatively, when any data change method of the database system is called, the changed data is recorded separately by presetting a cache item; wherein the presetting cache item includes at least one item in a message queue and a memory database.
[0043] Specifically, in engineering implementation, code implantation may cause performance degradation in the business system. Therefore, the embodiment of the present application introduces asynchronous or buffered implementation to better solve the performance degradation problem.
[0044] Specifically, when the update method of the database system is called, a background task, namely the Asynch task, is started to record the key-value pair key and value separately without affecting the continued execution of the update method. In addition, caching technologies such as message queues and in-memory databases can also be used to cache key and value data, improve the execution speed of the record method in the code, and reduce the time cost of capturing change data and the impact on the business system.
[0045] It should be noted that, in addition to the update operation for data updating, the embodiments of the present application are also applicable to the insert operation for data writing and the delete operation for data deletion. As long as any method is called, a background task can be started to cache the data.
[0046] Step 103: before returning the response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is successful, the separately recorded change data is captured.
[0047] In one embodiment of the present application, the data change result recorded in the response object is obtained through the injected interception code; wherein the data change result includes change success and other values. The data change result is analyzed, and when it is determined that the data change result is a successful change, the separately recorded change data is captured; wherein the change data includes at least text data and semi-structured data. When it is determined that the data change result is other values, the current data capture task is terminated.
[0048] Specifically, taking data update as an example, the update method provides the function of updating the key and value in the database. Therefore, if the entry parameters key and value of the update method can be recorded when update is called, the changes made to the database by this operation are captured, and the capture of the changed data key is completed.
[0049] Furthermore, the update method first establishes a database connection, and then uses the connection to issue a data update request to the database. Finally, the response resp object after the database update is completed is returned to the caller of the update. The resp object records whether the operation result of this update is successful. Therefore, if you want to obtain the changes made to the data in the database by update, the code can be changed to: before the return statement, if the value in resp is a successful operation, then record the key and value values. In the code, key and value are only operation objects for the database, which can be any type of data, including but not limited to: text, semi-structured data, etc.
[0050] Furthermore, by adding a judgment on the resp object value, if the database operation is successful, the resp object value is "OK", indicating that the value corresponding to the key has changed, and the key and value of the database change data can be recorded at this time. If the resp value is other, it means that the value has not been written, indicating that the data has not been successfully changed, so there is no need to record. In the sendOut method of the code, the recorded key and value data can be sent to any required system, such as a message queue, a target database system, etc.
[0051] The code in the embodiment of the present application only records the data to be operated without changing the interface function. Since the name and parameters of the update method have not changed, the execution mode of the business system has not changed compared to before the change.
[0052] For relational databases that support structured data, the entry parameters key and value of the update method may be specific SQL operation commands. Since the changed data also exists in the SQL command, the entire SQL (including the data contained in the SQL) can be captured directly to achieve the capture of changed data for the relational database system.
[0053] In one embodiment of the present application, when a data change request initiation command is received, the data corresponding to the data change command is uniquely identified and marked, and the marked data is cached. After receiving the data change request submission information, the marked data is sent to the corresponding target end. After receiving the data change request rollback information, based on the identification information in the rollback information, data query is performed in the cached data, and the queried data is deleted.
[0054] Specifically, general relational databases usually provide transaction operations. In this case, the operation may be rolled back for various reasons (such as user termination of business, abnormal operation of business system, network failure, etc.), and the data change will be canceled. Therefore, when the transaction is initiated, the operation command is first cached, and after the transaction is formally committed, this part of the cached data is sent out. If the transaction is rolled back, the cached data is discarded or deleted from the cache. At this time, it is necessary to record an ID in the cache for each opened transaction to uniquely mark the transaction. The subsequent data stored in the cache will be based on the ID. If the transaction is rolled back, the cached data corresponding to the ID can be discarded or deleted.
[0055] Step 104: Send the captured change data to the target end in the order of capture to complete synchronization of the change data.
[0056] In one embodiment of the present application, the data type corresponding to the marked data is determined, and a reference data set with the same data type is determined in a historical database. The cache duration corresponding to the data in the reference data set is determined, and the maximum cache duration is used as the first reference duration. The mean cache duration corresponding to multiple data in a preset period before receiving the marked data is determined in the historical database, and the mean cache duration is used as the second reference duration. The first reference duration and the second reference duration are weighted to obtain the data expiration duration. The marked data is marked for expiration time based on the data expiration time, and when the expiration time is reached, the marked data is deleted from the cache.
[0057] Specifically, the marked data is parsed to identify its key features, and the data features obtained by parsing are compared with the preset data type library. This type library contains definitions and descriptions of various data types, such as text, images, videos, structured data, etc. Through comparison, it can be determined to which type the marked data belongs. In the historical database, according to the determined data type, a reference data set of the same type is retrieved, which contains cache records of the same type of data in the past period of time. Analyze the data in the reference data set, for example, you can view the creation time, update time, and deletion time of the data from the cache. By calculating the time difference between these data points, the cache duration of each data can be obtained. The maximum value is selected from these cache durations as the first reference duration.
[0058] Furthermore, in the historical database, locate the preset time period (such as the last hour, day, etc.) before receiving the marked data. Within this time period, retrieve all data of the same type and their corresponding cache durations, and calculate the average of these cache durations as the second reference duration (cache mean duration).
[0059] Furthermore, according to business needs, corresponding weights are assigned to the first reference duration and the second reference duration, and these weights are used to perform weighted average calculation on the two durations to obtain the data expiration duration. Based on the calculated data expiration duration, an expiration time mark is set for the marked data. When this expiration time is reached, a deletion operation is triggered to delete the marked data from the cache.
[0060] The embodiment of the present application can dynamically determine the expiration time of data according to historical data and business needs, ensuring that the data in the cache remains fresh and valid. At the same time, it can also optimize the efficiency of cache usage and avoid waste of resources due to expired or invalid data.
[0061] In one embodiment of the present application, a timestamp is assigned to each captured change data, and a priority identification is performed based on the type of captured change data. The captured change data is cached in multiple data queues, and the multiple change data in each data queue are sorted based on the priority identification. The target timestamps corresponding to the multiple data queues are determined, and the target queue and the reference queue are determined based on the target timestamps; wherein the target timestamp is the timestamp of the latest data corresponding to the data queues. According to the sorting order, the change data in the target queue is sent to the target end, and during the sending process, the target timestamps corresponding to the target queue and the reference queue are compared. When the target timestamp corresponding to the reference queue is faster than the timestamp corresponding to the target queue, and the timestamp difference is greater than the preset duration, the reference queue is used as the new target queue to send the change data.
[0062] Specifically, for each captured change data, the system assigns a timestamp. This timestamp usually reflects the exact time when the data change occurred, which can be the system time when the data was captured or other relevant time stamps. At the same time, according to the type of captured change data, the system assigns it a priority identifier. This identifier can be determined based on the importance, urgency or other business logic of the data. For example, changes in critical business data may have a higher priority. The captured change data will be cached in multiple data queues. These queues can be set according to different business logic or processing requirements, such as according to data type, source or priority. In each data queue, the system sorts the change data according to the previously assigned priority identifier. In this way, when processing or sending data, high-priority data can be processed first.
[0063] For each data queue, its corresponding target timestamp is determined. Usually, it is the timestamp of the latest data in the queue, which represents the latest status of the queue data. Based on these target timestamps, the target queue and reference queue can be further determined.
[0064] Send the changed data in the target queue to the target end in the sorted order. This can be done through network transmission or other data transmission mechanisms. During the sending process, the target timestamps corresponding to the target queue and the reference queue are constantly compared to ensure the efficiency and accuracy of data sending. If it is found that the target timestamp corresponding to the reference queue is faster than the timestamp corresponding to the target queue, and the timestamp difference is greater than the preset duration, it means that the data in the reference queue is fresher or more important, then the reference queue is used as the new target queue for sending the changed data.
[0065] The captured change data is managed and processed through timestamps and priority tags to ensure that the data can be sent in order according to business needs and importance. At the same time, by dynamically adjusting the target queue, the system can more efficiently utilize the data resources in the cache and improve the efficiency of data processing and transmission.
[0066] Figure 2 A schematic diagram of the overall architecture comparison before and after the data interception code change provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, before the interception code is injected, the business system process includes a database access layer, and after the code is injected, the business system process includes a database access layer and a monitoring code layer. Figure 3 A schematic diagram of a change data capture operation process provided in an embodiment of the present application is as follows: Figure 3 As shown, after the user initiates a business, the business system executes the database operation code, writes the data into the database request, and determines whether the data is written successfully. If the data is written successfully, the changed data is recorded, the current task is ended, and the business initiated by the user is listened to again.
[0067] Figure 4 A change data capture summary diagram provided in an embodiment of the present application, such as Figure 4 As shown, after the user initiates the business, the business system executes the database operation code, the business system starts the transaction, the monitoring code generates a transaction ID, the business system changes the data, caches the changed data, and determines whether the transaction is successfully submitted. If the transaction is successfully submitted, the cached data is sent. If the transaction is not successfully submitted, the transaction is rolled back and the cached data is discarded.
[0068] Figure 5 This is a schematic diagram of the structure of a system change data capture device provided in an embodiment of the present application. Figure 5As shown, the system change data capture device 200 includes: at least one processor 201; and a memory 202 that is communicatively connected to the at least one processor 201; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 so that the at least one processor 201 can: obtain process information of the business system, determine the corresponding language type based on the process information, and inject interception code into the business system based on the language type and the preset programming interface; wherein the injected interception code is used to record the input parameter value of the preset programming interface; after the database system receives the data change request, the change data is recorded separately through the preset data cache task; before returning the response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured; in the order of capture, the captured change data is sent to the target end to complete the synchronization of the change data.
[0069] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to: obtain process information of a business system, determine a corresponding language type based on the process information, and inject interception code into the business system based on the language type and a preset programming interface; wherein the injected interception code is used to record the input parameter value of the preset programming interface; after the database system receives a data change request, the changed data is recorded separately through a preset data cache task; before returning a response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and when the data change result is a successful change, the separately recorded change data is captured; and the captured change data is sent to the target end in the order of capture to complete the synchronization of the change data.
[0070] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0071] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various changes and variations. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application.
Claims
1. A method for capturing system change data, characterized in that: The method comprises: Acquire process information of the business system, determine a corresponding language type based on the process information, and perform interception code injection on the business system based on the language type and a preset programming interface; wherein the injected interception code is used to record input parameter values of the preset programming interface; After the database system receives a data change request, it records the changed data separately through a preset data cache task; Before returning the response object generated after the database system is changed to the corresponding caller, the data change result recorded in the response object is obtained through the injected interception code, and if the data change result is a successful change, the change data recorded separately is captured; In the order of capture, the captured change data is sent to the target end to complete the synchronization of the change data; The intercepting code injection to the business system based on the language type and the preset programming interface specifically includes: Based on the language type, determine the code to be injected, and compile the code to be injected into a shared object file; Attaching a code injection tool to a service process corresponding to the business system, and obtaining register information corresponding to the service process; Based on the libc.so base address corresponding to the target process, the libc.so base address corresponding to the local process, and the required function address corresponding to the local process, determine the required function address corresponding to the service process; wherein the required function address includes at least one of the mmap address, the dlopen address, the dlsym address, and the dlclose address; Calling a required function, and injecting the shared object file into a service process corresponding to the business system based on the required function; The detach function is remotely called to detach from the target process, so that the service process corresponding to the business system continues to run, so as to complete the injection of the interception code.
2. A method for capturing system change data according to claim 1, characterized in that: The calling of the required function and injecting the shared object file into the service process corresponding to the business system based on the required function specifically includes: Call the mmap function to save the parameter information by applying for memory space; Writing the shared object file into the applied memory space, and remotely calling the dlopen function to open the written shared object file; Call the dlsym function to load the target function address corresponding to the service process, write the target function parameters, remotely call the target function and execute it; The dlclose function is remotely called to unload the shared object file and restore the original environment of the register.
3. A method for capturing system change data according to claim 1, characterized in that: After the database system receives the data change request, the changed data is recorded separately by presetting the data cache task, which specifically includes: When any data change method of the database system is called, a preset background asynchronous task is started to separately record the changed data; Alternatively, when any data change method of the database system is called, the changed data is recorded separately through a preset cache item; wherein the preset cache item includes at least one item in a message queue and a memory database.
4. A method for capturing system change data according to claim 1, characterized in that: After the database system receives the data change request, the method further includes: When receiving a data change request initiation command, uniquely identify the data corresponding to the data change command and cache the marked data; After receiving the data change request submission information, the marked data is sent to the corresponding target end; After receiving the data change request rollback information, based on the identification information in the rollback information, data query is performed in the cached data, and the queried data is deleted.
5. A method for capturing system change data according to claim 4, characterized in that: After caching the marked data, the method further includes: Determine the data type corresponding to the marked data, and determine a reference data set with the same data type in a historical database; Determine the cache duration corresponding to the data in the reference data set, and use the maximum cache duration as the first reference duration; Determine in the historical database the average cache duration corresponding to a plurality of data within a preset period before receiving the marked data, and use the average cache duration as a second reference duration; Performing weight calculation on the first reference duration and the second reference duration to obtain a data expiration duration; An expiration time is marked for the marked data based on the data expiration time, and when the expiration time is reached, the marked data is deleted from the cache.
6. A method for capturing system change data according to claim 1, characterized in that: The step of obtaining the data change result recorded in the response object by the injected interception code, and capturing the change data recorded separately when the data change result is successful, specifically includes: Obtaining the data change result recorded in the response object through the injected interception code; wherein the data change result includes change success and other values; Analyzing the data change result, and if it is determined that the data change result is successful, capturing the change data recorded separately; wherein the change data includes at least text data and semi-structured data; When it is determined that the data change result is the other value, the current data capture task is ended.
7. A method for capturing system change data according to claim 1, characterized in that: The captured change data is sent to the target end in the order of capture to complete the synchronization of the change data, specifically including: Assigning a timestamp to each captured change data and identifying a priority based on the type of the captured change data; Cache the captured change data into multiple data queues, and sort the multiple change data in each data queue based on the priority identifier; Determine target timestamps corresponding to the plurality of data queues respectively, and determine a target queue and a reference queue according to the target timestamps; wherein the target timestamps are timestamps of the latest data corresponding to the data queues respectively; According to the sorting order, the changed data in the target queue is sent to the target end, and during the sending process, the target timestamps corresponding to the target queue and the reference queue are compared; When the target timestamp corresponding to the reference queue is faster than the timestamp corresponding to the target queue, and the timestamp difference is greater than a preset duration, the reference queue is used as a new target queue to send the changed data.
8. A system change data capture device, characterized in that: The device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 7.
9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions can execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Capturing and processing interactions with a user interface of a native application
CN112074813A
Interface access log acquisition record query method and device and storage medium
CN114595201A
Data synchronization method, device and system based on Uliweb
CN116775756A
System and method for database replication by interception of in memory transactional change records
US20060047713A1
Systems and methods for debugging and application development
US20210357308A1
