Observable call link series connection method and system and storage medium

By obtaining and processing the preset format logs or transaction details generated by the system, a log containing global identification and extended information is created, which realizes accurate, comprehensive and rapid call link connection in complex scenarios, and solves the existing technology The problem of link interruption and difficulty in evaluating accuracy improves operation and maintenance monitoring efficiency.

CN119025366APending Publication Date: 2024-11-26UNIONPAY INT CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310601717.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing call link series technology is difficult to achieve accurate, comprehensive and fast link series when facing complex real scenarios, especially external system interactions and nodes that have not completed log transformation, resulting in link interruptions and result accuracy. Difficult to evaluate.

Method used

By obtaining the preset format log or transaction details generated by the system, create a log containing the global identity, current node identity, parent node identity and extension information, form a call link in series, and issue a reminder when an exception is detected to achieve the implementation of the Accurate, comprehensive and fast link connection.

Benefits of technology

It effectively solves the problem of external system interaction and link series of unfinished log transformation nodes, provides accurate, comprehensive and fast system call link series, and improves the efficiency of operation and maintenance monitoring and fault location capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025366A_ABST
    Figure CN119025366A_ABST
Patent Text Reader

Abstract

The invention provides a method for observably calling link series connection, and particularly provides an accurate and rapid log link series connection scheme for different business scenes that transactions are interactively processed in internal and external systems and an internal system cannot generate logs in a preset format. The method specifically comprises the steps that logs generated according to a preset format when all application systems in a first system process data are obtained, the logs with the same global identifier are selected from the obtained logs, and the logs are connected in series to form a calling link according to current node identifiers and father node identifiers of all the logs and system processing time; under the condition that the data is processed by a second system interacting with the first system, logs with partial key field content missing in the second system are connected in series on the basis of extension fields of the logs; and under the condition that the first system cannot generate the log in the preset format, constructing the log in the preset format based on the transaction detail pipeline data, and connecting the log in series to the calling link. The invention also provides a link anomaly detection method and a corresponding system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information technology, and more specifically, to observability call link serialization technology. Background Art

[0002] Digital transformation is an inevitable trend in the current development of enterprise technology. It aims to use modern technology and communication methods to break through data barriers between different levels and industries, improve the overall operating efficiency of the industry, and build a new digital economic system. The digital transformation of operation and maintenance capabilities is also gradually developing from traditional system-oriented operation and maintenance services to business scenario-oriented operation and maintenance services. The refinement of operation and maintenance scenarios, the digitization of operation and maintenance capabilities, and the intelligence of operation and maintenance technologies are the main development trends in the future.

[0003] Observability, which is to deduce the internal state of a system by analyzing the data generated by the system, is a new set of ideas and methodologies for the field of operation and maintenance monitoring. Gartner defines observability as a feature of software and systems that allows administrators to collect data about the external and internal state of the system so that they can answer questions about its behavior. I&O, DevOps, and SRE teams then use this data to investigate anomalies, participate in observability-driven development, and improve system performance and uptime. Metrics, logs, and traces are the three pillars of system observability. Metrics are used to discover and perceive problems, traces are used to locate where there are problems, and logs are used to analyze why problems occur. Observability can effectively improve the efficiency of system monitoring. Metrics, logs, and traces are the three pillars of observability and are inseparable from each other. From discovering indicator anomalies to drilling down layer by layer to trace the fault points of the link, and finally viewing the specific error logs, the full-link automated root cause location can be achieved.

[0004] Currently, indicator and log monitoring have been widely used in actual operation and maintenance scenarios, while call link technology is still in the exploratory stage. Call link is one of the main technologies for fault monitoring and root cause location of abnormal events during operation and maintenance. By tracking the execution process of a single user request in various business systems, efficient diagnosis and location of system faults can be achieved. With the promotion of microservice architecture, further refinement of service granularity, rapid increase in the number of service applications, and increasingly complex calls between services, the demand for call link technology is also increasing.

[0005] Although the current Internet and financial industries are vigorously promoting the application of call link technology in the field of operation and maintenance, in fact, there are differences in the digital foundations of enterprises. Enterprises need to comprehensively consider these current situations in the process of upgrading and transforming their systems, and then formulate adjustment plans.

[0006] The current mainstream call link concatenation technology is mainly based on Dapper, a complex distributed link tracing solution proposed by Google. It provides a data model design that is conducive to problem location, application-level transparent measurement practices, and a low-overhead design solution, solving some problems of link tracing in industrial applications. Since the Google Dapper paper was proposed, many developers have been inspired by it and proposed various link tracing solutions, such as Twitter's Zipkin and Naver's Pinpoint in 2012, Wu Sheng's Skywalking in 2015, and Uber's Jaeger. Most of these frameworks are based on the principles of Dapper and use the link transformation specifications of OpenTracing. On this basis, some modules have been improved and different functions have been integrated.

[0007] The above technology is an ideal framework for serializing call links. However, the actual scenarios are relatively complex. There are problems such as the need for the internal system to interact repeatedly with the external system, which leads to interruptions in the call link; some application systems have not completed log transformation for a long time, which leads to interruptions in the call link; and it is difficult to evaluate the accuracy of the results of the serialization of links. Therefore, targeted improvements are needed to the serialization process of the links. Summary of the invention

[0008] According to one aspect of the present application, a method for observability call link concatenation is provided to solve at least one of the problems raised above.

[0009] According to the observability call link concatenation method of the present application, it includes: obtaining a log of a preset format generated when each application system in the first system processes data, or when there is an application system in the first system that cannot generate a log of a preset format, obtaining the transaction detail flow data generated by the application system and creating a log of a preset format based on it, wherein the log includes a global identifier, a current node identifier, a parent node identifier, a system processing time, and extended information; selecting logs with the same global identifier from the logs of the preset format, and concatenating them into a call link based on the global identifier, current node identifier, parent node identifier, and system processing time of each log; wherein the extended information includes basic fields and optional fields, and the basic fields include a field indicating the transaction time. time, business serial number, and organization identification field, the optional information includes a destination identification field and the optional field is configured to assign a value to the destination identification field to indicate the identity of the next processing node when the next processing node cannot generate a log in a preset format; wherein, when the data is processed by a second system that interacts with the first system, a log that is different from the global identifier of the initial call link but the same as the basic fields in the extended information of each log in the initial call link is selected from the acquired logs as a re-selected log; the position of each log in the re-selected log in the call link is determined according to the current node identifier of each log in the re-selected log, and the re-selected log is connected in series to the call link according to the determined position.

[0010] The observability call link concatenation method, optionally, in the case where there is an application system processing in the first system that cannot generate a log in a preset format, obtaining the transaction detail flow data generated by the application system and creating a log in a preset format includes: creating a log in a preset format including extended information based on the transaction time, the unique business serial number requested, and the institution identifier indicated in the transaction detail flow data; determining the current node identifier and the parent node identifier of the created log based on the constructed extended information and the extended information of each log in the call link.

[0011] The observability call link concatenation method may optionally further include: modifying the global identifier of the reselected log to be the same as the global identifier of each log in the call link.

[0012] The observability call link concatenation method may optionally further include using the global identifier of the call link as the global identifier of the created log that conforms to a preset format.

[0013] The observable call link concatenation method may optionally further include performing one or more of the following tests on the call link: text variable detection, numerical variable anomaly detection, and link sequence anomaly detection.

[0014] The observability call link concatenation method may optionally further include issuing an exception reminder when an exception is detected.

[0015] Provided is an observable call link serial connection system, comprising: a first unit, which is used to obtain logs generated by each application system in a first system when processing data, wherein the logs are generated according to a preset format, and the logs generated according to the preset format include a global identifier, a current node identifier, a parent node identifier, and extended information; a second unit, which is used to select logs with the same global identifier from the obtained logs and form a preliminary call link; a third unit, which is used to determine the position of each log in the preliminary call link based on the current node identifier and the parent node identifier of each log in the preliminary call link, and to set the position of each log in the preliminary call link according to the determined position, thereby forming a final call link; wherein the extended information includes basic fields and optional fields, wherein the basic fields include fields indicating transaction time, business serial number, and institution identifier, and the optional information includes a destination identifier field, and the optional field is configured to assign a value to the destination identifier field to indicate the identity of the next processing node when the next processing node is unable to generate a log according to the preset format.

[0016] In the system of observability call links connected in series, optionally, the first unit is configured to select, from the acquired logs, a log that is different from the global identifier of the preliminary call link but has the same basic fields as the basic fields in the extended information of each log in the preliminary call link, as a re-selected log when the data is processed by a second system that interacts with the first system; the third unit is configured to determine the position of each log in the re-selected log in the preliminary call link according to the current node identifier of each log in the re-selected log, and to associate the re-selected log with the call link based on the determined position.

[0017] In the system in which observability call links are connected in series, optionally, the first unit is configured to obtain transaction detail flow data generated by an application system that cannot generate a log in a preset format in the first system when the data is processed by the application system in the first system; the second unit is configured to establish a log in a preset format based on the transaction time, business serial number, and institution identification indicated in the transaction detail flow data, and the log includes extended information constructed based on the transaction time, business serial number, and institution identification indicated in the transaction detail flow data, and is associated with the preliminary call link based on the constructed extended information; the third unit is configured to determine the current node identifier and the parent node identifier in the constructed log based on the optional fields in the extended information of each log in the preliminary call link; and associate the established log in the preset format with the call link based on the determined current node identifier and parent node identifier.

[0018] The observable call link serialization system optionally further includes a fourth unit for performing one or more of the following detections on the call link: text variable detection, numerical variable anomaly detection, and link sequence anomaly detection.

[0019] The observability call link series system may optionally further include a fifth unit for issuing an abnormality reminder when an abnormality is detected.

[0020] A computer-readable storage medium is also provided, on which one or more programs are stored, the one or more programs including instructions, the instructions being executable by one or more processors to implement the observability call link concatenation method described above. As an example, the computer-readable storage medium is a non-transitory readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present application will be more fully understood by referring to the following detailed description of the specific implementation modes in conjunction with the accompanying drawings.

[0022] Figure 1 It is a flowchart of the observability call link concatenation method according to the example of the present application;

[0023] Figure 2 It is a simple schematic diagram of the process of processing a service request initiated by a user through different service systems of the first system according to an example of the present application;

[0024] Figure 3 Indicated according to Figure 1 The observability call chain shown forms a call chain of methods connected in series;

[0025] Figure 4A It is a schematic diagram of the process of processing business when the first system interacts with the second system;

[0026] Figure 5 is a flowchart of a method for observability call link concatenation in an example of interaction with a second system according to an example of the present application;

[0027] Figure 4B is in accordance with Figure 5 The observability call chain shown is a concatenated method pair Figure 4A The logs shown are connected in series to form a schematic diagram of the call chain;

[0028] Figure 4C The process from 302-2C to 302-2C' is shown;

[0029] Fig. 6A The process of processing data by the first system 302 is illustrated;

[0030] Figure 6B It illustrates the process of the transaction details flow data record changing to log D' and then to log D" when the application system 5 processes business F;

[0031] Figure 7 is a flowchart of a method for observability calling link concatenation in an example where log transformation has not yet been completed according to the present application;

[0032] Figure 8 It is a schematic diagram of the structure of the observability call link series system of the present application;

[0033] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an example of the present application. DETAILED DESCRIPTION

[0034] In order to help those skilled in the art to accurately understand the subject matter for which protection is sought in the present application, the specific implementation of the present application will be described in detail below in conjunction with the accompanying drawings.

[0035] This application is based on the OpenTracing specification and monitors different types of businesses in the system separately in a business-oriented manner, so as to clearly display the complete link of a certain type of business from initiation to the background core system to meet the fine-grained monitoring needs. This application also considers how to assign a unified global serial number to the same business request when it repeatedly interacts with external nodes within the application system to connect the links in series, and how to use network traffic data or transaction flow data to supplement when some application system logs are not completely transformed. Finally, a monitoring method is proposed for the call link quality obtained by connection, which can achieve accurate, comprehensive and fast application system call link connection.

[0036] Combined with more complex real-world scenarios, it is difficult to achieve comprehensive observability call link concatenation in at least two scenarios. One is the scenario of repeated interaction with external systems, and the other is the existence of nodes in the internal system that have not completed the log record transformation.

[0037] In the actual process of link series connection, due to business needs, transactions must interact repeatedly between the internal system (equivalent to the first system) and the external system (equivalent to the second system), such as: some financial transactions need to interact with the external regulatory system, and after the external system completes the transaction processing, the response message is transmitted back to the internal system. At this time, since the application log of the external system cannot be obtained and the external system has not performed log transformation (that is, the log is not recorded in the preset format described in this application), it is impossible to assign a unified global serial number (that is, global identification) to the transaction like the internal system, resulting in the global serial number in the response message returned by the external system being inconsistent with the previous request message. Calling link series connection based on the observability of this application will effectively solve this problem and provide accurate, comprehensive and fast system call link series connection.

[0038] The transformation of internal system logs is a long process. Depending on the size of the enterprise, it often involves dozens or even hundreds of core or edge application systems within the enterprise. It is not easy to quickly build a set of observable call link solutions and achieve a smooth transition during the long log transformation process. Since the data of the table structure such as network traffic or transaction flow of the application system (node) that has not completed the transformation in the internal system is inconsistent with the data structure of the log type, it needs to be processed before it can be constructed into a series call link together with the logs of other application systems. By adopting the method of serializing observable call links according to the example of this application, such processing can be completed so that the logs of the application system that has not completed the transformation can be easily constructed into the series call chain.

[0039] In addition, according to the data processing method of the present application, it also includes performing one or more of the following detections on the constructed call link: text variable detection, numerical variable anomaly detection and link sequence detection.

[0040] Various examples according to the present application will be described in detail below with reference to the accompanying drawings.

[0041] Figure 1It is a flow chart of a data processing method according to an example of the present application. As shown in the figure, in step S100, the log generated when each application system in the first system processes data is obtained, and the log is generated according to a preset format. The log generated according to the preset format includes a global identifier, a current node identifier, a parent node identifier, a system processing time, and extended information; or, in the case where there is an application system in the first system that cannot generate a log in a preset format, the transaction detail flow data generated by the application system is obtained and a log in a preset format is created accordingly. In step S102, a log with the same global identifier is selected from the obtained logs and a preliminary call link is formed. In step S104, based on the global identifier, current node identifier and parent node identifier, system processing time and other information included in each log in the selected log, the position of each log in the preliminary call link is determined. In step S106, according to the determined position of each log in the preliminary call link, a call link with a certain logical order is connected in series. The preliminary call link is to initially associate logs with the same global identifier, while the final call link is to connect the logs in the preliminary call link in logical order through information such as the global identifier, current node identifier, parent node identifier, and system processing time to construct an observable call link.

[0042] Take the first system as a financial payment system as an example. A user initiates a payment transaction through an electronic terminal such as a mobile phone. The financial payment system receives the payment transaction request initiated by the user, and different application systems of the financial payment system process the payment transaction respectively. It should be understood that the payment transaction request has been converted into machine-readable data when processed by the device, so this application also refers to it as data. Each application system will record the processing log in a preset format during the process of processing data. Figure 1 By processing the data in the process shown, the log for the transaction can be constructed as a call link, thereby facilitating the operation and maintenance of the financial payment system. For example, when an abnormality occurs during operation, the root cause of the abnormality can be easily determined based on the call link, and the cause of the abnormality can also be determined accordingly.

[0043] According to the examples of the present application, the extended information includes basic fields and optional fields. The basic fields include fields representing transaction time, business serial number, and institution identification. The optional fields include a field for destination identification. The optional fields are configured to assign a value to the destination identification field to indicate the identity of the next application system when the next application system that processes data is a system that cannot generate logs in a preset format.

[0044] The first system may be a system for processing business data, such as the financial payment system mentioned above. The application systems within the first system are subsystems used to enable the first system to achieve its functions, or in other words, each application system is a component of the first system. According to the example of the present application, the global identifier included in the log is used to identify the data within the first system so that when the data is processed by any application system in the first system, the identity of the data can be known based on the global identifier. Each application system of the first system will generate a log when processing data. For example, when application system A processes data, it generates a log of application system A processing the data, and when application system B processes the data, it generates a log of application system B processing the data, and so on.

[0045] Figure 2 It is a simple schematic diagram of the process of processing a service request initiated by a user according to the example of this application through different service systems of the first system. Figure 2 In the example, the user initiates a service request. As is known, the service request is converted by the processing device into machine-readable data (e.g., binary data) and enters the first system. In the first system, it is processed by application system 1, application system 2, application system 3, application system 4, and application system 5 in sequence. In order to achieve the function that the service request is to achieve in the first system, each application system processes the data differently to achieve different functions. In order to distinguish the data processed by each system, the data processed by each application system is marked differently, for example Figure 2 The data 1, data 2, data 3, data 4, data 5, data 6, data 7, data 8 and data 9 shown in FIG. It should be noted that each application system can be implemented on the same device or on different devices. The device implementing the application system can be a computer, tablet, smart phone and other electronic devices with data processing capabilities.

[0046] Each application system of the first system is configured to record logs in a preset format. In some cases, there are application systems in the first system that may not be able to generate logs in the preset format, such as internal systems that have not been completely transformed. In contrast, application systems that can generate logs in the preset format are application systems that have been completely transformed. The logs in the preset format include the following fields: global identifier TraceID, current node identifier SpanID, parent node identifier ParentSpanID, system processing time, and extended information Extend. The global identifier TraceID is a business (for example Figure 2The unique identification number used to mark the identity of the business (data) during the processing of the first system (a business request initiated by a user) is included in the log for the business. Therefore, the data of the business after being processed by different application systems can be indexed according to the global identifier. The data processed by the application system is, for example, transaction requests, response messages and other data. The current node identifier SpanID is the node identifier of each application system as a processing node when the business is processed in the first system. The parent node identifier ParentSpanID is the previous processing node of the processing node represented by the current node identifier SpanID. The extended information Extend is a JSON string in this application, including basic fields and optional fields. The basic fields include the transaction time TransTm, the business serial number TraceNum, and the institution identifier IIN field. The optional fields include the destination identifier Des field.

[0047] Table 1 shows the key fields and corresponding descriptions included in the log required by the preset methods of some examples of this application.

[0048]

[0049] As shown in the fields in Table 1, the identifier OrigServ of the first processing node, the current node identifier SpanID, and the parent node identifier ParentSpanID can be used to convey the calling relationship between the parent and child nodes. The timestamp StartTime can be used to calculate the overall time of the current system processing transactions, or to determine the order of each interaction in the process of repeated interactions between the two systems, so as to distinguish between request and response messages. The extended information Extend can contain multiple subfields, and each subfield can be configured according to actual needs to include some auxiliary information useful for link series connection, or configure some important business information, such as the transaction time TransTm, business serial number TraceNum, institution identifier IIN mentioned above, and in some cases also include the next node identifier Des, etc. In the example of this application, the global identifier is the identity identifier of the business in the first system. Once the business processing involves an external system, the global identifier of the log generated by the application system of the first system for the external system processing will change; the business serial number is only related to the business processing process. Regardless of whether the business processing passes through the external system, it is the serial number for the business. Usually, the global identifier is an explicit number that can be obtained in the log without decryption, while the business serial number may require decryption of the system message to obtain.

[0050] For example, a user may initiate a payment request to a financial payment system such as the first system, and each application system of the financial payment system will process the payment request. During the payment process, the financial payment system may need to interact with an external supervision system such as the second system.

[0051] Figure 3 Indicated according to Figure 1 The call link formed by the observable link series method shown in FIG. 3 is a call link formed by the observable link series method shown in FIG. 3. The business enters the first system 302 and is processed by each application system in the first system 302. Figure 1 and Figure 3 Obtain (step S100) logs generated when each application system in the first system 302 processes data, each log including a global identifier. According to the global identifier in the obtained log, the logs with the same global identifier ( Figure 3 In the example, the log with TraceID of 000001 is formed (step S102) as a preliminary call link. In this example, the preliminary call link has a TraceID of 000001, indicating that the logs therein are for the same business. For the logs of the preliminary call link, the order of arrangement of each log in the preliminary call link is determined according to the current node identifier, parent node identifier, and system processing time in each log. For example, the current node identifier SpanID in log 302-2 is 2, and the value of the parent node identifier ParentSpanID is 1, indicating that it is data processed after the application system 1 whose current node identifier SpanID is 1. Therefore, according to the time sequence of processing data, log 302-2 is after log 302-1. The current node identifier SpanID in log 302-3 is 3, and the value of the parent node identifier ParentSpanID is 2, indicating that it is data processed after the application system 2 whose current node identifier SpanID is 2. According to the time sequence, log 302-3 is after log 302-2. The current node identifier SpanID in log 302-4 is 4, and the parent node identifier ParentSpanID is 2, indicating that it is data processed after application system 2 whose current node identifier SpanID is 2. In chronological order, log 302-4 is after log 302-2. Therefore, if Figure 3 As presented, log 302 - 1 , log 302 - 2 , log 302 - 3 , and log 302 - 4 form a serial call chain in the chronological order of processing the business.

[0052] Figure 4A This is a schematic diagram of the process of processing business when the first system interacts with the second system. The business, application system 1, and application system 2 in this diagram are all related to Figure 3As shown, the difference is that the application system 2 interacts with the second system 300 to process the business, for example, the financial payment system in the above example interacts with the external regulatory system. As shown in Figure 4, the application system 2 initiates a processing request 304 to the second system 400, and the second system 400 gives feedback 305. For this purpose, the application system 2 generates logs 302-2B and 302-2C. Log 302-2B is a log generated when data is processed in the first system 302, so the global identification number is 000001, while log 302-2C is the processing of the external system recorded by the application system 2. At this time, its global identification number 000002 is different from the global identification number in the first system 302. Therefore, log 302-2C cannot be associated with the call link with a global identification of 000001 according to its global identification number. According to the illustration of the present application, each application system within the first system 302 is configured so that when it is necessary to interact with the second system 400 as an external system, the application system that sends data to the second system 400 should indicate the identification number of the second system 400 as the external system in the extended information field of its log. Specifically, the organization number of the second system 400 is assigned as a value to the destination identification field which is an optional field.

[0053] Figure 5 A flowchart illustrating a method for serializing observability calls in interaction with a second system. Figure 4B is in accordance with Figure 5 The observability call chain shown is a concatenated method pair Figure 4A The log 302-2C shown is a schematic diagram of a call link constructed to the global identifier 000001, which is similar to Figure 4A The difference is, Figure 4B The extended information field of the log is also shown.

[0054] It should be noted that in each example of the present application, since the preliminary call link is formed by a log with the same global identifier, the same global identifier is also referred to as the global identifier of the preliminary call link; in the case where the preliminary call link behavior is a call link, the global identifier is also referred to as the global identifier of the call link.

[0055] This will be combined with Figure 5 and Figure 4BTo illustrate. In step S500, the logs generated when each application system in the first system processes data are obtained. For example, logs 302-1, 302-2B and 302-2C are obtained. In step S502, logs with the same global identifier are selected from the obtained logs and a preliminary call link is formed. For example, logs 302-1 and 302-2B with a global identifier of 000001 are selected from logs 302-1, 302-2B and 302-2C, and they are associated with each other to form a preliminary call link. In step S504, based on the current processing node and parent node identifiers included in each log in the selected log, the position of each log in the preliminary call link is determined. In step S506, a series call link is formed according to the determined position. For example, according to the current node identifier 2 and the parent node identifier 1 in the log 302-2B, it can be known that it is behind the application system 1 which is the previous processing node according to the time sequence of processing data, so the log 302-2B is set behind the log 302-1 in the preliminary call link. During the data processing process, an interaction occurs between the first system 302 and the external system (i.e., the second system 400). According to the example of the present application, the method will execute step S508 to select from the acquired logs a log that is different from the global identifier of the preliminary call link, but the basic fields in the log extended information are the same as the basic fields in the extended information in the preliminary call link. The basic fields of the extended information include the transaction time TransTm, the business serial number TraceNum, and the institution identifier IIN. For example, for Figure 4BThe business shown has a transaction time of 20221010, a business serial number of 123456, and an institution identifier of 00010000. Log 302-2C is selected as the reselected log because it has the same extended information as the preliminary call link. In step S510, the position of each log in the preliminary call link formed in step S504 is determined according to the current node identifier of each log in the reselected log 302-2C, or alternatively, the position of each log in the call link formed in step S506 is determined according to the current node identifier of each log in the reselected log. And in step S512, the reselected log is associated with the call link formed in step S506 according to the determined position. Specifically, the reselected log is set at a suitable position in the call link according to the chronological order of processing data. For example, the current node indicated in the reselection log 302-2C is 2, indicating that it is a log generated by the application system 2 when processing data. Combined with the value of the destination identification field Dest in 302-2B, it can be known that the reselection log 302-2C is the feedback of the second system 400 to the application system 2, and the feedback is the feedback 305 for the processing request 304 initiated by the application system 2 to the second system 400. More specifically, the value of the destination identification field Dest indicates that the next node of 302-2B is the second system 400, indicating that the data will enter the external system. In this application, by associating the value of the extension field of the received 302-2C message with the value of the extension field of 302-2B, it can be known that 302-2C is feedback from the second system 400. Through the above process, the position of the reselection log 302-2C in the preliminary call link can be more accurately confirmed, and the reselection log 302-2C can be associated with the call link in the order of the determined processing time. According to an example of the present application, the method further includes modifying the global identifier of the reselected log 302-2C to the global identifier of the preliminary associated data, for example, modifying the global identifier 000002 of the reselected log 302-2C to 000001, thereby forming log 302-2C′, Figure 4C The process from 302 - 2C to 302 - 2C′ is illustrated.

[0056] Combination of the above Figure 4A , Figure 4B , Figure 5 as well as Figure 4CTaking the described process as an example, before the adoption of this application, when a request message is sent from an internal system to an external system, the global identifier of the log is different from the global identifier in the log when the response message of the external system returns from the external system to the internal system again, and it cannot be connected in series with the original transaction. After adopting the observability call link connection method in the method example of this application, the log of the response message of the external system when it returns from the external system to the internal system again can be constructed into a log with basically the same format as the log of the internal system and with the same global identifier based on the relevant data of the application system of the associated internal system, so that the logs can also be connected in series in this case, achieving effective improvement of the link connection process and providing accurate, comprehensive and fast system call link connection.

[0057] If there is an application system in the first system 302 that cannot generate logs in the preset format, then according to the example of the present application, a log in the preset format including extended information can be generated based on the non-transaction detail flow data recorded by the application system, such as transaction time, business flow number, institution identification, etc. This situation occurs when a system or node in the first system has not been transformed into a system or node that generates logs in the preset format.

[0058] Fig. 6A The process of constructing the transaction detail flow data of the application system 5 that cannot generate the preset format into a log in the preset format in the first system 302 is illustrated. According to the present application, when the application system 5 cannot generate the log in the preset format, as the upstream processing node that sends the processing request to the application system 5, the application system 1 should mark the application system 5 as the next processing node in the extended field of its log. Specifically, the application system 1 assigns the node identifier of the application system 5 as a value to the optional field Dest of the extended field of the application system 1.

[0059] like Fig. 6A As shown, the log 302-5D of the application system 5 includes logs generated for different businesses and including basic information, such as the business serial number TraceNum 651, the institution identifier IIN 652, and the transaction time TransTm 653.

[0060] like Fig. 6A As shown, application system 1 and application system 6 are logs of the same service F, and their global identification TraceID is 000001; the basic fields in the extended information Extend have the same contents, namely: TraceNum is 111111, IIN is 00010000, and TransTm is 20221010.

[0061] According to the content in the basic field of the extended information Extend of application system 1 and application system 6, the log related to business F can be found from the transaction details flow data 302-5D of application system 5, that is, the first line 600B of the log, whose content is the same as the content in the basic field of the extended information Extend of application system 1 and application system 6. Based on the log content recorded in the first line 600B of application system 5, the log of application system 5 processing business F is constructed according to the preset format. The constructed log is shown as D', including the global identifier TraceID, the current node identifier SpanID, the parent node identifier ParentSPanID, and the extended information Extend in the preset format, wherein the field value of the uncertain value is empty. According to the value of dest in the log of application system 1, it can be known that application system 5 is the next processing node of application system 1, thereby the values ​​of the current node identifier SpanID and the parent node identifier ParentSPanID in D' can be known. At the same time, the global identifier 000001 of the call link generated by application system 1 and application system 6 processing business F can be used as the value of the global identifier of log D", thereby constituting log D". Figure 6B The diagram illustrates the process of the transaction details flow data record changing to log D' and then to log D" when the application system 5 processes business F.

[0062] Figure 7 is a flowchart of a method for observability call link concatenation according to some examples of the present application, wherein the first system includes an application system that cannot generate logs in a preset format, such as an application system that has not yet completed log transformation. Figure 7 and Fig. 6A In step S708, the transaction details flow data of the application system that cannot generate the log in the preset format in the first system is obtained, and the transaction flow record is generated according to the data, for example Fig. 6A In step S710, a log in a preset format is generated based on the information indicating the transaction time, the unique serial number requested, and the organization identification in the transaction details data. The log includes extended information constructed based on the information indicating the transaction time, the unique serial number requested, and the organization identification in the transaction details data. For example, Fig. 6A In step S714, the current node identifier and the parent node identifier in the constructed log are determined according to the optional fields in the extended information of each log in the preliminary call link. For example, from the extended information in the constructed log D', the current node identifier and the parent node identifier in the constructed log are determined according to the optional fields in the extended information of each log in the preliminary call link. Fig. 6A The log D' to Fig. 6AIn step S716, the log in the preset format is associated with the call link according to the determined current node identifier and parent node identifier. Fig. 6A , the call link is a call link formed by logs A and E of application system 1 and application system 6. Alternatively, in step S712 and step S714, the value of the field in the log to be constructed can also be determined according to the information in the call link determined in step S706.

[0063] Figure 7 In the example, the process of establishing the call link between the logs A and E of the application system 1 and the application system 6 is as follows: Figure 1 The method shown is constructed. For example, logs in a preset format generated when each application system in the first system processes data are obtained; logs with the same global identifier are selected from the obtained logs to form a preliminary call link; based on the current processing node and parent node identifier included in each log in the selected logs, the position of each log in the preliminary call link is determined; the position of each log in the preliminary call link is set according to the determined position, thereby forming a call link with a certain logical order.

[0064] If the transformation of the internal system log is not completed, the transaction details of the untransformed application system can be constructed into a log in a preset format according to the observability call link concatenation method of the example of this application, and then concatenated into the call link. Fig. 6A , Figure 6B and Figure 7 The described method is used to match the detailed record D in the network traffic or transaction flow in the original log of the unreformed application system 5, and convert the detailed record into a log format D′ according to, for example, the log of the application system 1. In the data middle platform module, all indicator type data are converted into a simple log format to facilitate log aggregation. It can be found that the extend field of log D′ is the same as the corresponding field content in the detailed record D, but the traceId, spanId, parentSpanId and other information of log D′ are empty. Then further aggregation is performed based on the content of the sub-fields of traceId or extend, so that log A, log D′ and log E are aggregated together. In order to better connect log D′ with the previous and next context logs in series, it can be known that the next node is application system 5 based on the content of the dest field in the log of application system 1. Then, the log D′ on the application system 5 is further converted into D″, whose traceId is the traceId of log A, spanId is itself, and parentSpanId is the spanId of log A. Thus, the logs of the application system 1, log D″, etc. are finally connected in series according to the conventional transaction link connection process to obtain a complete call chain.

[0065] According to the data processing method of each example of the present application, it also includes performing one or more of the following detections on the constructed call link: text variable detection, numerical variable anomaly detection and link sequence detection.

[0066] Simply put, log anomaly detection can be divided into log variable anomaly detection and link sequence anomaly detection; among them, text variable detection is used to detect whether the fields of the log are normal, data variable detection is used to detect whether the distribution of the values ​​of each field is normal, and link sequence anomaly detection is used to detect whether there are any abnormalities in the sequence of the constructed call links.

[0067] Text variable detection

[0068] For text type variables, the BloomFilter algorithm can be used to detect whether there are anomalies. BloomFilter is a fast search algorithm that maps multiple hash functions. It is usually used in some situations where it is necessary to quickly determine whether an element belongs to a set, but it does not require 100% correctness. Bloomfilter uses a hash function to map an element to a point on an array of length m, for example. When this point is 1, it indicates that the element is in the set, otherwise it is not in the set. When the detected element is often, k hash functions can be used to correspond to k points. If all points are 1, then the detected element is in the set. If there is a 0, the corresponding element is not in the set.

[0069] As described above, the log of the present application includes multiple fields, such as a global identifier TraceID, a current node identifier SpanID, etc. These fields are detected by using, for example, a Bloom Filter algorithm.

[0070] Data variable detection

[0071] In the call chain composed of logs, the values ​​of the fields of each log are numerical values. KS (Kolmogorov–Smirnov) can be used to check whether the distribution of numerical values ​​is abnormal.

[0072] For variables of numerical type, anomalies can be detected by comparing the distribution of training set and test set. Generally, the appearance of individual extreme values ​​does not mean that there is an error in link series connection, but the overall change of numerical distribution may mean transaction anomaly. When the distribution of test data is significantly different from that of training data, that is, when the KS test rejects the null hypothesis, it is considered that there is an anomaly.

[0073] Link sequence anomaly detection

[0074] The abnormal detection of the link sequence of the call link in the log can be judged by comparing the difference between the current link sequence and the sequence in the historical link sequence library. When the current link sequence does not match any sequence in the link sequence library or hits a sequence that appears very rarely in the process, it is considered that an abnormality may have occurred. Abnormal situations can be classified into the following four types:

[0075] Error: Compared with the normal sequence, the positions of a few nodes are wrong;

[0076] Redundant: Compared with the normal sequence, there are some extra nodes at a few positions;

[0077] Deletion: Compared with the normal sequence, nodes are missing at a few positions;

[0078] Rare: hits a sequence that is rarely seen in the linked sequence library.

[0079] As an example, the BlAST algorithm may be used to compare the current link sequence with the historical link sequence.

[0080] Figure 8 Schematic diagram of the structure of the observability call link series system according to the present application. Figure 8 As shown, the processing system includes a first unit 800, a second unit 802, a third unit 804 and a fourth unit 806, and an optional fifth unit 808 represented by a dotted line.

[0081] The first unit 800 is configured to obtain logs in a preset format generated when each application system in the first system processes data. The logs generated in the preset format include a global identifier, a current node identifier, a parent node identifier, a system processing time, etc., and also include extended information. The second unit 802 is configured to select logs with the same global identifier from the obtained logs and form a preliminary call link. The third unit 804 is configured to determine the position of each log in the preliminary call link based on the global identifier, current node identifier, parent node identifier, and system processing time of each log in the preliminary call link, and form a final call link in series based on the determined position. The processing system can be used to execute the above combined Figure 1 For example, the first unit 800 is used to execute step S100, the second unit 802 is used to execute step S102, and the third unit 804 is used to execute step S104 and step S106.

[0082] The first unit 800 is also configured to select, from the acquired logs, a log that is different from the global identifier of the preliminary call link but has the same basic fields as the basic fields in the extended information of each log in the preliminary call link, as a reselected log when the data is processed by the second system that interacts with the first system. The third unit 804 is configured to determine the position of each log in the reselected log in the preliminary call link according to the current node identifier of each log in the reselected log, and to connect the reselected log in series to the call link according to the determined position. This processing can be used to perform, for example, Figure 5 The method described.

[0083] The first unit 800 is also configured to obtain transaction detail flow data generated by an application system in the first system that cannot generate a log in a preset format when the data is processed by the application system in the first system, and perform the following process to create a log in a preset format: based on the fields indicating the transaction time, business serial number, and institution identification in the transaction detail flow data, establish a log that conforms to the preset format, and the log includes extended information constructed based on the fields indicating the transaction time, business serial number, and institution identification in the transaction detail flow data.

[0084] The third unit 804 is configured to determine the current node identifier and the parent node identifier in the constructed log according to the optional fields in the extended information of each log in the preliminary call link; and associate the established log in the preset format with the call link based on the determined current node identifier and parent node identifier.

[0085] The fourth unit 806 is used to perform one or more of the following detections on the call link: text variable detection, numerical variable anomaly detection, and link sequence anomaly detection. In some examples, the data system according to the example of the present application also includes a fifth unit 806, which will issue an alarm when an anomaly is detected.

[0086] According to the example of this application, for example, Figure 8 The processing system described may be implemented as a software program, hardware, or a combination of both. The processing system may be implemented, for example, in conjunction with the first system 302 (see Figure 3 ) can also be implemented in an electronic device that communicates with the first system 302 (such as a computer, a tablet, a smart phone, a cloud server), or in a device that implements the first system 302.

[0087] According to an example of the present application, an electronic device is also provided. Fig. 9900, a processor 902, and a memory 904. The processor 902 may be one or more. The memory 904 stores programs, which include instructions for implementing the various data processing methods described above in conjunction with the figures and examples. In other words, when the electronic device is used to communicate with the first system 302 (see Figure 3 ) and / or the second system 400 (see Figure 4) communicate through the communication device 900, or are implemented in the first system 302, during the process of the first system 302 processing business data, the processor 902 executes the program in the memory 904 to implement the various data processing methods described above.

[0088] A computer-readable storage medium is also provided, on which one or more programs are stored, wherein the one or more programs include instructions that can be executed by one or more processors to implement the observability call link concatenation method described in any of the examples above.

[0089] Although specific embodiments of the present application have been shown and described in detail to illustrate the principles of the present application, it should be understood that the present application can be implemented in other ways without departing from such principles.

Claims

1. A method for observability call link concatenation, characterized in that: The method comprises: Obtaining a log in a preset format generated when each application system in the first system processes data, or when there is an application system in the first system that cannot generate a log in a preset format, obtaining transaction detail flow data generated by the application system and creating a log in a preset format based on the data, wherein the log includes a global identifier, a current node identifier, a parent node identifier, a system processing time, and extended information; Select logs with the same global identifier from the logs in the preset format, and concatenate them into a call link based on the global identifier, current node identifier, parent node identifier, and system processing time information of each log; The extended information includes a basic field and an optional field, the basic field includes a field indicating transaction time, business serial number, and institution identification, the optional field includes a destination identification field, and the optional field is configured to assign a value to the destination identification field to indicate the identity of the next processing node when the next processing node cannot generate a log according to a preset format; Wherein, in the case where the data is processed by a second system interacting with the first system: From the obtained logs, a log having a different global identifier from the preliminary call link but the same basic fields as the basic fields in the extended information of each log in the preliminary call link is selected as a re-selected log; The position of each log in the reselected log in the call link is determined according to the current node identifier of each log in the reselected log, and the reselected log is serially connected to the call link according to the determined position.

2. The observability call link concatenation method according to claim 1, characterized in that: In the case that an application system that cannot generate a log in a preset format exists in the first system, obtaining transaction detail flow data generated by the application system and creating a log in a preset format includes: Creating a log in a preset format including extended information according to the transaction time, business serial number, and institution identification in the transaction detail flow data; According to the constructed extended information and the extended information of each log in the call link, the current node identifier and the parent node identifier of the created log are determined.

3. The observability call link concatenation method according to claim 1, characterized in that: The method further comprises: The global identifier of the reselected log is modified to be the same as the global identifier of each log in the call link.

4. The observability call link concatenation method according to claim 1, characterized in that: The method also includes using the global identifier of the call link as the global identifier of the created log that complies with a preset format.

5. The observability call link concatenation method according to any one of claims 1 to 4, characterized in that: The call link is also tested by one or more of the following methods: Text variable detection, Anomaly detection for numerical variables, and Link sequence anomaly detection.

6. The observability call link concatenation method according to claim 5, characterized in that: The method further comprises: When an abnormality is detected, an abnormality reminder is issued.

7. An observability call link tandem system, comprising: The first unit is used to obtain logs in a preset format generated when each application system in the first system processes data, or when there is an application system in the first system that cannot generate a log in a preset format, obtain transaction detail flow data generated by the application system and create a log in a preset format based on the log, wherein the log generated in the preset format includes a global identifier, a current node identifier, a parent node identifier, a system processing time, and extended information; The second unit is used to select logs with the same global identifier from the acquired logs and form a preliminary call link; A third unit is used to determine the position of each log in the preliminary call link based on the global identifier, current node identifier, parent node identifier, and system processing time of each log in the preliminary call link, and to connect the logs in series into a final call link according to the determined position; Among them, the extended information includes basic fields and optional fields, the basic fields include fields representing transaction time, business serial number, and institution identification, the optional fields include a destination identification field, and the optional fields are configured to assign a value to the destination identification field to indicate the identity of the next processing node when the next processing node cannot generate a log in a preset format.

8. The observability call link serialization system according to claim 7, characterized in that: The first unit is configured to select, from the acquired logs, a log that is different from the global identifier of the preliminary call link but has the same basic fields as the basic fields in the extended information of each log in the preliminary call link, as a re-selected log when the data is processed by the second system interacting with the first system; The third unit is configured to determine the position of each log in the preliminary call link according to the current node identifier of each log in the reselected log, and to connect the reselected log in series to the call link according to the determined position.

9. The observability call link serialization system according to claim 7, characterized in that: The first unit is configured to, when the data is processed by an application system in the first system that cannot generate a log in a preset format, obtain transaction detail flow data generated by the application system, and perform the following process to create a log in a preset format: Creating a log in a preset format including extended information according to the transaction time, business serial number, and institution identification in the transaction detail flow data; According to the constructed extended information and the extended information of each log in the call link, the current node identifier and the parent node identifier of the created log are determined.

10. The observability call link serialization system according to any one of claims 7 to 9, characterized in that: The system further comprises: The fourth unit is configured to perform one or more of the following detections on the call link: Text variable detection, Anomaly detection for numerical variables, and Link sequence anomaly detection.

11. The observability call link cascading system according to claim 10, characterized in that: The system further comprises: The fifth unit is used to issue an abnormality reminder when an abnormality is detected.

12. A computer-readable storage medium, characterized in that: One or more programs are stored on the storage medium, and the one or more programs include instructions, which can be executed by one or more processors to implement the observability call link serialization method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Access request marking method, device, equipment and storage medium

    CN109474698A

  • Data processing method and device

    CN111598683A

  • Service calling information generation method and device and storage medium

    CN112905353A