Database-based full-link tracking method and device, storage medium and product

By using link tracing mechanism and log processing mechanism in the database, we can associate the application link log and data change log, and the generation of full-link tracing logs is achieved, solving the problem that data change logs cannot be recorded in the existing technology, ensuring that Link and data integrity.

CN119046103APending Publication Date: 2024-11-29CHINA MERCHANTS BANK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411228786.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing full-link tracking technology cannot record data change logs, resulting in incomplete links and data being tracked.

Method used

By using the link tracing mechanism to track the call link of the database by external applications, and recording the data changes of the database in combination with the log processing mechanism, and linking the application link log and data change log to achieve the generation of the full link tracing log.

Benefits of technology

The data change log is recorded and the data changes before and after the service change is tracked, so that the link and data can be unified into one log, solving the problem of link incompleteness in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046103A_ABST
    Figure CN119046103A_ABST
Patent Text Reader

Abstract

The invention discloses a full-link tracking method and device based on a database, a storage medium and a product, and relates to the technical field of distributed storage. The full-link tracking method based on the database comprises the steps that a link tracking mechanism is used for tracking a calling link of an external application program to the database, and an application link log is obtained; a log processing mechanism is used for recording the data change condition of the database, a data change log is obtained, and the data change condition is generated when the database responds to the calling link; and associating the application link log with the data change log according to the data change condition to obtain a full-link tracking log. According to the technical scheme provided by the invention, the problem that the tracked link and data are incomplete due to the fact that a full-link tracking technology provided by the existing scheme cannot change according to data before and after the business change can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a database-based full-link tracking method, device, storage medium and product. Background Art

[0002] Full-link tracing is a link tracing tool used to monitor and analyze requests in distributed storage systems. The tool can track the request flow path and integrate monitoring data to achieve end-to-end data monitoring. Through full-link tracing technology, problems can be quickly located, monitoring performance can be optimized, and user experience can be improved.

[0003] At present, full-link tracking technology is developing rapidly in the process of Internet microservice development. Existing full-link tracking technology is mainly based on methods such as log collection, data statistics and analysis to achieve monitoring and optimization of user behavior and system performance. For example, the prior art provides a data full-link tracking method and system based on a message bus. The data full-link tracking method includes the following steps: consumers obtain tracking messages; consumers publish tracking messages to the Kafka message bus; the Kafka message bus sends tracking messages to the big data service; if the big data service determines that the tracking message is information that meets the preset conditions, the big data service sends subscription information to the subscription service through the Kafka message bus.

[0004] The above-mentioned full-link tracking solution can realize full-link tracking and monitoring of the entire data processing process in a distributed system. However, the current full-link tracking technology mainly records the application's call link. Changes in application behavior usually lead to changes in business data. Therefore, when some business data changes, it is necessary to record the data changes before and after the change. However, the existing full-link tracking mechanism mainly records the application's call link and cannot track the data changes before and after the business change. This results in incomplete data tracked by the full-link tracking technology.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a database-based full-link tracking method, device medium and product, aiming to solve the technical problems that the full-link tracking technology provided by the existing technical solutions cannot record data change logs and the tracked links and data are incomplete.

[0007] To achieve the above objectives, the present application provides a database-based full-link tracking method, including:

[0008] Use the link tracking mechanism to track the call link of the external application to the database and obtain the application link log;

[0009] Use the log processing mechanism to record the data changes in the database and obtain the data change log, where the data changes are generated by the database response call link;

[0010] According to the data change situation, the application link log and the data change log are associated to obtain the full link tracking log.

[0011] Preferably, in the above-mentioned full-link tracing method, the step of using the link tracing mechanism to trace the call link of the external application to the database and obtaining the application link log includes:

[0012] Get the call request from the external application to the database;

[0013] Use the tracing frontend of the link tracing mechanism to inject tracing tags into the call request;

[0014] Use tracking tags to track the transfer process of call requests along the call link and record application link logs in real time;

[0015] When the call request is transmitted along the call link, the tracking backend using the link tracking mechanism sends the call request carrying the tracking mark to the database; at the same time, the tracking backend is used to upload the application link log to the log processing center.

[0016] Preferably, in the above-mentioned full-link tracking method, the step of using a log processing mechanism to record data changes in the database and obtaining a data change log includes:

[0017] Add a tracking mark field to the data table in the database, and add a data mapping annotation to the corresponding persistent object in the database, where the data mapping annotation corresponds to the data table;

[0018] Use SQL interceptor to intercept call requests and extract tracing tags from the call requests;

[0019] Using SQL Interceptor, trace tags are written into the trace tag field according to the data mapping annotation.

[0020] Preferably, in the above-mentioned full-link tracking method, the step of using a log processing mechanism to record data changes in the database and obtaining a data change log includes:

[0021] Use a log processing mechanism to monitor data changes in the database, where the data changes include tracking tags;

[0022] Use data mapping annotations to record data changes and obtain data change logs;

[0023] The tracking mark is searched from the tracking mark field corresponding to the data change situation, and the tracking mark is written into the data change log.

[0024] Preferably, in the above-mentioned full-link tracking method, the step of associating the application link log and the data change log according to the data change situation to obtain the full-link tracking log includes:

[0025] Use the Kafka mechanism to subscribe to application link logs and data change logs;

[0026] Match the tracking tags of the application link log and data change log;

[0027] When the tracking tags of the application link log and the data change log match successfully, the tracking tags are used to associate the application link log and the data change log to obtain the full link tracking log.

[0028] Preferably, in the above-mentioned full-link tracking method, the step of associating the application link log and the data change log according to the data change situation to obtain the full-link tracking log includes:

[0029] Retrieve metadata information about data changes from the metadata information repository corresponding to the database;

[0030] Combine metadata information, application link logs, and data change logs to build a full-link tracking log.

[0031] Preferably, in the above-mentioned full-link tracking method, the step of using a log processing mechanism to record data changes in the database and obtaining a data change log includes:

[0032] Use the binlog mechanism to monitor data changes in the database;

[0033] When monitoring the database for data changes, back up the data in the database and obtain the data backup log;

[0034] Parse the data backup log to obtain the data before the database change;

[0035] Generate a data snapshot of the data before the change.

[0036] To achieve the above objectives, the present application also provides a database-based full-link tracking device, the device including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the database-based full-link tracking method provided in any of the above technical solutions.

[0037] To achieve the above objectives, the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the database-based full-link tracking method provided by any of the above technical solutions are implemented.

[0038] To achieve the above objectives, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the database-based full-link tracking method provided by any of the above technical solutions.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] The technical solution provided by the embodiment of the present application uses a link tracking mechanism to track the call link of the external application to the database, thereby obtaining an application link log; combined with the log processing mechanism to record the data changes in the database, thereby obtaining a data change log, wherein the data changes are generated by the corresponding call link of the database; therefore, combined with the data change situation, it is possible to associate the above-mentioned application link log and data change log, and unify the data change log and the application link log into the full-link tracking log, thereby realizing the full-link connection of the log. In addition, the log processing mechanism is a processing mechanism that comes with the database, and its link is clear. The logical processing process of the log is based on out-of-process calculation and has no effect on the system. In summary, the technical solution provided by the embodiment of the present application can record the data change log, track the data changes before and after the business change, so that the tracked links and data can be unified and combined into one log. The above method can solve the technical problem that the full-link tracking technology provided by the prior art cannot record the data change log, and the tracked links and data are incomplete. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of the process flow of the first database-based full-link tracking method provided in an embodiment of the present application;

[0044] Figure 2 For this application Figure 1A flowchart of a method for generating an application link log is provided;

[0045] Figure 3 For this application Figure 1 A flowchart of a first method for generating a data change log is provided;

[0046] Figure 4 For this application Figure 1 A flowchart of a second method for generating a data change log is provided;

[0047] Figure 5 for Figure 1 A flowchart of the first method for generating a full-link tracking log is provided;

[0048] Figure 6 yes Figure 1 A flowchart of the second method for generating a full-link tracking log is provided;

[0049] Figure 7 It is a flowchart of a second database-based full-link tracking method provided in an embodiment of the present application;

[0050] Figure 8 It is a flowchart of a third database-based full-link tracking method provided in an embodiment of the present application;

[0051] Fig. 9 It is a structural diagram of a database-based full-link tracking device provided in an embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The existing technology has the following technical problems: the existing full-link tracking solution, the existing full-link tracking technology is mainly based on log collection, data statistics and analysis methods to monitor and optimize user behavior, system performance and other aspects. However, the current full-link tracking technology can achieve full-link tracking and monitoring of the entire data processing process in a distributed system. Changes in application behavior usually lead to changes in business data. Therefore, when some business data changes, it is necessary to record the data changes before and after the change. However, the existing full-link tracking mechanism mainly records the application call link and cannot track the data changes before and after the business change. This results in the data tracked by the full-link tracking technology being incomplete.

[0056] To solve the above technical problems, the present application provides a solution by associating application link logs and data update logs with data change conditions to obtain a full-link tracking log, which can not only track the external application's call link to the database, but also obtain the database's data changes, so that the tracked links and data can be complete.

[0057] In summary, the database-based full-link tracking solution provided by this application can solve the problem that the full-link tracking technology in the prior art cannot record data change logs and the tracked links and data are incomplete.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, a computer-readable storage medium, etc.

[0059] In order to better understand the above technical solution, the exemplary embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments shown here. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0060] To achieve the above objectives, the present application provides a database-based full-link tracking method, including:

[0061] S110: Use the link tracing mechanism to track the call link of the external application to the database and obtain the application link log. The link tracing mechanism here can use the opentracing mechanism. OpenTracing is a set of distributed tracing protocols that provides a unified interface that is independent of the platform and language, making it convenient for developers to integrate one or more distributed tracing implementations in their own services. Through this link tracing mechanism, the call link of the external application to the database can be tracked to obtain the application link log. The external application is relative to the database and can be in the form of an application for the front-end display interface and an external client.

[0062] Specifically, as a preferred embodiment, Figure 2 As shown, in the above-mentioned full-link tracing method, step S110: using the link tracing mechanism to track the call link of the external application to the database, and obtaining the application link log step includes:

[0063] S111: Obtain a call request from an external application to a database; the call request usually includes a link path of a call link, which defines the call location of the relevant data in the database; therefore, the embodiment of the present application can determine the call link through the call request by obtaining the call request from an external application to a database.

[0064] S112: Using the tracking front-end of the link tracking mechanism, the tracking tag is injected into the call request; as mentioned above, the call request contains a link path of a call link, including the call location of the relevant data, so by injecting the tracking tag into the call request, the link path passed can be recorded.

[0065] S113: Use the tracking mark to track the transmission process of the call request along the call link, and record the application link log in real time. The call request contains the link path of the call link, so when the tracking front end sends the call request, it can be transmitted along the path of the call link. Therefore, using the tracking mark contained in the call request, each node of the call link passed can be recorded, thereby recording the application link log of the call link in real time.

[0066] S114: When the call request is transmitted along the call link, the tracking backend using the link tracking mechanism will send the call request carrying the tracking mark to the database, and use the tracking backend to upload the application link log. The call request contains the call location in the database where the data needs to be called. Therefore, when the call request is transmitted along the call link, the tracking backend using the link tracking mechanism can send the call request to the database, and the database can extract the data at the corresponding location by parsing the call request. When the call link is transmitted, the tracking backend will upload the above-mentioned application link log to the log processing center for processing. In addition, when the call request is sent to the database, the tracking mark will enter the database, so that when the data in the database changes accordingly, the tracking mark can be recorded in the field corresponding to the changed data, thereby matching the tracking mark with the data change situation, which is convenient for subsequent recording of the data change situation in the database.

[0067] Combination Figure 8 It can be seen from the illustrated embodiment that the tracing front end of the link tracing mechanism (specifically, the application front end of the hardware system) sends a call request carrying the traceID to the tracing back end. After receiving the call request, the tracing back end checks whether it carries the traceID. If the traceID cannot be found, it is necessary to further generate the traceID, and then write the traceID into the application link log. At the same time, the call request is passed to the database, so that the traceID can be transmitted to the corresponding field in the database, thereby recording the data changes in the database.

[0068] Figure 1 The technical solution provided by the illustrated embodiment further includes the following steps after obtaining the application link log:

[0069] S120: Use a log processing mechanism to record data changes in the database to obtain a data change log, wherein the data change is generated by the database in response to the call link.

[0070] In the technical solution provided by the embodiment of the present application, the log processing mechanism can record the data changes of the database, thereby obtaining a data change log. The log processing mechanism here can be a database binlog processing mechanism, which is a database field-level mechanism, so it can log and summarize the data changes in the database to obtain a data change log. Since the log logic processing process of the mechanism is based on out-of-process calculation, it will not affect the system. The binlog processing mechanism has its own monitoring function, which can spit out the corresponding data change log when the data changes.

[0071] Specifically, as a preferred embodiment, Figure 3As shown, in the above-mentioned full-link tracking method, step S120: using a log processing mechanism to record data changes in the database, and obtaining a data change log step includes:

[0072] S121: Add a tracking mark field to the data table of the database, and add a data mapping annotation to the corresponding persistent object of the database, wherein the data mapping annotation corresponds to the data table. The data mapping annotation is used to mark the changed data and record the data changes before and after the data change. Because the above data change situation corresponds to the tracking mark, the data mapping annotation here can write the above tracking mark into the tracking mark field in the data table when recording the data change situation.

[0073] S122: Use the SQL interceptor to intercept the call request and extract the tracking mark from the call request. The SQL interceptor intercepts the call request, and because the call request contains the tracking mark, the SQL interceptor will intercept the tracking mark.

[0074] S123: Using the SQL interceptor, write the tracking mark into the tracking mark field corresponding to the call request according to the data mapping annotation. The SQL interceptor can intercept the call request, and the call request carries the tracking mark. The data mapping annotation can correspond to the data table. When the data in the data table changes, that is, when the above data changes, the corresponding position of the data table will change. In the tracking mark field of the data table, the tracking mark can be written into the tracking mark field using the data mapping annotation.

[0075] Combination Figure 8 As can be seen from the process shown, the embodiment of the present application intercepts the update request based on the data mapping annotation, then obtains the traceID from the application, and finally modifies the update SQL log, so that the traceID can be written into the updated SQL log using the SQL statement.

[0076] In the technical solution provided by the embodiment of the present application, the data mapping annotation corresponds to the data table and can record the data change situation. In this way, the data mapping annotation can determine the data table and field corresponding to the data change situation. After the SQL interceptor is applied to intercept the tracking mark, the data mapping annotation can be used to write the tracking mark into the tracking mark field corresponding to the data change situation. When the binlog processing mechanism is used to generate a data change log, the corresponding tracking mark field can be queried from the tracking mark field corresponding to the data change situation, so that the tracking mark can be further written into the data change log. Because the data change situation is generated by the database in response to the above-mentioned call link, and the tracking mark corresponds to the data change situation, the tracking mark can be used to establish an association relationship between the application link log and the data change log.

[0077] In addition, as a preferred embodiment, Figure 4 As shown, in the above-mentioned full-link tracking method, step S120: using a log processing mechanism to record data changes in the database, and obtaining a data change log step includes:

[0078] S124: Using the log processing mechanism to monitor the data changes in the database. The present application combines the database field-level binlog mechanism to perform log aggregation, so as to monitor the data changes in the database.

[0079] S125: Use data mapping annotations to record data changes and obtain data change logs. Data mapping annotations correspond to data tables, so that data changes will be reflected in the data tables, and data mapping annotations are added to POs of SQL operations. POs are persistent objects and will not be deleted due to data changes. Therefore, calling the data mapping annotations can determine specific data changes, thereby obtaining data change logs.

[0080] S126: extracting the tracking mark from the tracking mark field corresponding to the data change, and writing the tracking mark into the data change log. As mentioned above, when the data is changed, the data mapping annotation will write the tracking mark into the tracking mark field corresponding to the data change. In this way, by searching the tracking mark field corresponding to the data change from the data table, the tracking mark can be extracted and written into the data change log.

[0081] Combination Figure 8 From the process shown, we can see that the data mapping annotation can write the traceID into the business field. In this way, when using binlog monitoring, after monitoring the data changes in the database, we can find the business field corresponding to the data change, and then find the traceID and write it into the data change log.

[0082] The technical solution provided by the embodiment of the present application can use the tracking mark to establish an association relationship between the data change log and the above-mentioned application link log by extracting the tracking mark from the tracking mark field corresponding to the data change situation and writing the tracking mark into the data change log. Because the tracking mark corresponds to the data change situation, and the data change situation is recorded in the data change log and caused by the application link call, the tracking mark can establish an association relationship between the above-mentioned two logs.

[0083] Figure 1 The technical solution provided by the embodiment shown uses a log processing mechanism to record the data changes in the database and obtain a data change log.

[0084] S130: Associating the application link log and the data change log according to the data change situation to obtain a full-link tracking log. The data change situation is generated by the database response to the call link, and the data change log records the data change situation. In this way, the corresponding application link can be associated with the application link log by using the data change situation, thereby establishing an association relationship between the application link log and the data change log, thereby obtaining a full-link tracking log. The embodiment of the present application uses a tracking tag corresponding to the data change situation to associate the above two logs, because the tracking tag can not only mark the application link situation and save it in the application link log, but also mark the data change situation and write it into the data change log. In this way, the tracking tag can be used to associate the above application link log and the data change log, thereby obtaining a full-link tracking log.

[0085] Specifically, as a preferred embodiment, Figure 5 As shown, in the above-mentioned full-link tracking method, step S130: according to the data change situation, the step of associating the application link log and the data change log to obtain the full-link tracking log includes:

[0086] S131: Use the Kafka mechanism to subscribe to the application link log and data change log; Kafka is a high-throughput distributed publish-subscribe message system, which is widely used in the field of big data real-time processing. This application designs a log processing center, which can use the Kafka mechanism to subscribe to the above application link log and data change log by consuming two Kafka messages.

[0087] S132: Matching the tracking marks of the application link log and the data change log. As described above, the tracking mark corresponds to the data change status, and is recorded not only in the application link log but also in the data change log. Because the data change status is generated by the database in response to the above application call, the relationship between the two logs can be established by using the tracking mark.

[0088] S133: When the tracking tags of the application link log and the data change log match successfully, the tracking tags are used to associate the application link log and the data change log to obtain a full-link tracking log.

[0089] The technical solution provided by the embodiment of the present application is that the application link log is sent through the Kafka mechanism, and the binlog log of the database is also sent through the Kafka mechanism. In this way, a log processing center is built, and the log processing center can consume two Kafka messages at the same time through the Kafka mechanism. Then, the application link log and the data change log are associated with the tracking tag, so that the full-link tracking log can be obtained. In this way, the data change log and the application operation log can be unified into the full-link tracking log to realize the full-link connection of the log.

[0090] In addition, as a preferred embodiment, Figure 6 As shown, in the above-mentioned full-link tracking method, step S130: according to the data change situation, the step of associating the application link log and the data change log to obtain the full-link tracking log includes:

[0091] S134: Retrieving metadata information about data changes from a metadata information repository corresponding to the database;

[0092] S135: Combine metadata information, application link logs, and data change logs to build a full-link tracking log.

[0093] The technical solution provided by the embodiment of the present application is that the database corresponds to a metadata information library, which stores metadata information. The metadata information records some characteristic information of the data in the database. In this way, the metadata information is used to supplement the above-mentioned application link log and data change log, so that the changes before and after the data change in the database can be recorded, thereby obtaining a full-link tracking log.

[0094] In summary, the technical solution provided by the embodiment of the present application uses a link tracking mechanism to track the call link of the external application to the database, thereby obtaining an application link log; combined with the log processing mechanism to record the data changes in the database, thereby obtaining a data change log, wherein the data changes are generated by the corresponding call link of the database; therefore, combined with the data change situation, it is possible to associate the above-mentioned application link log and data change log, and combine the data change log and the application link log into the full-link tracking log, thereby realizing the full-link connection of the log. In addition, the log processing mechanism is a processing mechanism that comes with the database, and its link is clear. The logical processing process of the log is based on out-of-process calculation and has no effect on the system. In summary, the technical solution provided by the embodiment of the present application can record the data change log and track the data changes before and after the business change, so that the tracked links and data can be unified and combined into one log. The above method can solve the technical problem that the full-link tracking technology provided by the prior art cannot record the data change log, and the tracked links and data are incomplete.

[0095] In addition, in the embodiment of the present application, after the data in the database is changed, the data in the database may be backtracked, so it is necessary to back up the data before the change in the database. Specifically, as a preferred embodiment, Figure 7 As shown, after the above step S120: using the log processing mechanism to record the data changes in the database and obtaining the data change log, the method further includes:

[0096] S201: Use the binlog mechanism to monitor data changes in the database;

[0097] S202: when monitoring that there is a data change in the database, back up the data in the database and obtain a data backup log;

[0098] S203: Parse the data backup log to obtain the data before the change of the database;

[0099] S204: Generate a data snapshot of the data before the change.

[0100] The embodiment of the present application uses the binlog mechanism to monitor the data changes in the database, and backs up the data in the database when the data changes are detected, so that the data in the database can be backed up before the data changes to obtain a data backup log. When it is necessary to trace back data or format data later, by parsing the data backup log, the data before the change of the database can be obtained, and a data snapshot of the data before the change can be generated, thereby achieving the purpose of tracing back data or formatting data.

[0101] In addition, as a preferred embodiment, the present application also provides a database-based full-link tracking method, which is applied to a full-link tracking implementation system. The full-link tracking implementation system includes a link log collection module, an SQL execution interceptor, a full-link log collection module, and a log aggregation module; wherein,

[0102] Link log collection module: It can record through the existing link tracing opentracing mechanism and generate traceID. Specifically, the tracing front end adds traceID to the call request. If the tracing back end detects that the call request does not carry traceID, it needs to generate another traceID.

[0103] SQL execution interceptor: When used in business, it is necessary to add a field traceID in the database table, and add annotations to the persistent object PO operated during SQL execution; the annotation marks the data and records the data changes before and after the database table. The annotation can be matched with the traceID, and the tranceID is written to the specific traceID field. When SQL is executed, the function of the SQL execution interceptor is implemented based on Mybatis (a Java-based persistence layer framework). The SQL interceptor can intercept the call request, obtain the traceID from the above call request when SQL is executed, and query the traceID field corresponding to the data change through SQL rewriting, and write the traceID to the traceID field of the database. It should be noted that the traceID at the corresponding position in the traceID field marks the data change corresponding to the position. Subsequently, traceID is called from the traceID and written to the data change log.

[0104] Full-link log collection module: Application link logs are sent through Kafka, and database binlog logs are also sent through Kafka. Therefore, this application will build a full-link log collection module to consume two Kafka messages at the same time.

[0105] Log aggregation module: associate the logs on both sides through traceID Since both the application link log and the database binlog log contain traceID, the two logs can be associated using traceID. The traceID of the binglog log can be queried from the traceID field, and the metadata information of the database can be obtained to complete the description of the data change, thereby constructing a complete log.

[0106] Figure 8 The full-link tracking method shown specifically includes:

[0107] S301: The tracking front end sends a call request carrying the tranceID to the tracking back end.

[0108] S302: The tracing backend checks whether the call request carries the traceID; if not, a traceID is generated.

[0109] S303: Tracking backend, connecting to BeiDou system and BStack data structure;

[0110] S304: After connecting to the Beidou system, a Beidou trace log is generated through the BStack data structure.

[0111] S305: The tracing backend transmits the traceID to the database, specifically writes the tranceID in the update request, and then transmits the update request to the database.

[0112] S306: Intercept the update request based on the annotation.

[0113] S307: Obtain traceID from the application.

[0114] S308: Modify and update the SQL, and write the traceID into the updated SQL, so that the traceID can be added to the traceID business field in the data table.

[0115] S309: The binlog processing mechanism monitors the database to detect whether there is any data update in the database.

[0116] S310: When the binlog processing mechanism detects that there is data update in the database, it records the binlog log.

[0117] S311: Use the Kafka mechanism to subscribe to the above Beidou trace log and binlog log, and use these two logs and database metadata to build user operation logs.

[0118] S312: Use the operation log for query and audit.

[0119] S313: parse the binlog data before modification.

[0120] S314: Generate a data snapshot.

[0121] S315: Use the data snapshot to query and restore the database.

[0122] In summary, the traceID is written to the database through the SQL interceptor in the application, and the full-link operation log is implemented by integrating the database Binlog log and the application link log. It is completely separated from the business logic and does not affect the specific business of the database and the call link.

[0123] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the database-based full-link tracking method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0124] In addition, the database-based full-link tracking device provided in the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the database-based full-link tracking method of any of the above-mentioned embodiments.

[0125] Reference below Fig. 9 , which shows a schematic diagram of the structure of a database-based full-link tracking device suitable for implementing the embodiment of the present application. The database-based full-link tracking device in the embodiment of the present application can include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The database-based full-link tracking device shown is merely an example and should not bring any limitations to the functions and scope of use of the embodiments of the present application.

[0126] like Fig. 9As shown, the database-based full-link tracking device can include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can operate the database-based full-link tracking device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a model building device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0127] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0128] The database-based full-link tracking device provided in the present application, compared with the prior art, has the same beneficial effects as the database-based full-link tracking method provided in the above-mentioned embodiment, and other technical features in the database-based full-link tracking device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0129] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

[0131] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the database-based full-link tracking method in the above-mentioned embodiment.

[0132] The computer-readable storage medium provided in the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0133] The above-mentioned computer-readable storage medium can be included in the model building device of the database-based full-link tracking system; it can also exist independently without being assembled into the database-based full-link tracking device.

[0134] The computer-readable storage medium carries one or more programs, and when the one or more programs are executed by the model building device, the computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes also be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0136] The modules involved in the embodiments of the present application can be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0137] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned database-based full-link tracking method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the database-based full-link tracking method provided in the above-mentioned embodiment, and will not be repeated here.

[0138] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned database-based full-link tracking method.

[0139] Compared with the prior art, the beneficial effects of the computer program product provided in the present application are the same as the beneficial effects of the database-based full-link tracking method provided in the above-mentioned embodiment, and will not be elaborated here.

[0140] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A full-link tracking method based on a database, characterized in that: include: Use the link tracking mechanism to track the call link of the external application to the database and obtain the application link log; Using a log processing mechanism to record data changes in the database to obtain a data change log, wherein the data change is generated by the database in response to the call link; The application link log and the data change log are associated according to the data change situation to obtain a full-link tracking log.

2. The method according to claim 1, characterized in that The step of using the link tracking mechanism to track the call link of the external application to the database and obtain the application link log includes: Obtaining a call request from an external application to the database; Using the tracing front end of the link tracing mechanism, injecting a tracing tag into the call request; Using the tracking mark to track the transmission process of the call request along the call link, and recording the application link log in real time; When the call request is transmitted along the call link, the tracking backend of the link tracking mechanism is used to send the call request to the database, and the application link log is uploaded using the tracking backend.

3. The method according to claim 2, characterized in that The step of using a log processing mechanism to record data changes in the database and obtain a data change log includes: Adding a tracking mark field in a data table of the database, and adding a data mapping annotation in a persistent object corresponding to the database, wherein the data mapping annotation corresponds to the data table; intercepting the call request using a SQL interceptor, and extracting the tracking tag from the call request; Using the SQL interceptor, the tracking flag is written into the tracking flag field according to the data mapping annotation.

4. The method according to claim 3, characterized in that The step of using a log processing mechanism to record data changes in the database and obtain a data change log includes: Use a log processing mechanism to monitor data changes in the database; Using the data mapping annotation to record the data change situation, to obtain the data change log; The tracking mark is extracted from the tracking mark field corresponding to the data change status, and the tracking mark is written into the data change log.

5. The method according to claim 4, characterized in that The step of associating the application link log and the data change log according to the data change situation to obtain a full-link tracking log includes: Use the Kafka mechanism to subscribe to the application link log and the data change log; Matching tracking tags of the application link log and the data change log; When the tracking tags of the application link log and the data change log match successfully, the tracking tags are used to associate the application link log and the data change log to obtain the full-link tracking log.

6. The method according to claim 1, characterized in that The step of associating the application link log and the data change log according to the data change situation to obtain a full-link tracking log includes: Retrieving metadata information about the data change from a metadata information repository corresponding to the database; The full-link tracking log is constructed by combining the metadata information, application link log and data change log.

7. The method according to claim 1, characterized in that The step of using a log processing mechanism to record data changes in the database and obtain a data change log includes: Use the binlog mechanism to monitor data changes in the database; When monitoring that there is a data change in the database, back up the data in the database to obtain a data backup log; Parsing the data backup log to obtain the data of the database before the change; Generate a data snapshot of the data before the change.

8. A full-link tracking device based on a database, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the database-based full-link tracking method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the database-based full-link tracking method as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the database-based full-link tracking method as claimed in any one of claims 1 to 7 are implemented.