A transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry

Abstract

The present invention relates to the technical field of transaction tracking, and specifically to a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry, including flow collection, flow analysis and transaction concatenation. The present invention does not require message standardization transformation of non-standard systems, saves manpower and capital costs for system transformation, and avoids business continuity risks that may be caused by system transformation; secondly, the Jaccard similarity coefficient is used to compare the similarity between upstream and downstream messages, and self-discovery of non-standard transaction paths is achieved in an unsupervised manner without manual intervention; in addition, conventional concatenation, expert model rule concatenation, and intelligent matching concatenation are combined, and in scenarios where partial similarity matching cannot achieve satisfactory results, expert models can be used for characteristic matching, which is suitable for transaction concatenation of most non-standardized messages.

Description

Technical Field

[0001] The present invention relates to the technical field of transaction tracking, and in particular to a method and system for analyzing the entire transaction link based on non-standardized transaction messages in the banking industry. Background Art

[0002] In recent years, with the vigorous development of the banking IT industry, the scale of services has increased by orders of magnitude compared to traditional methods, and the business models have become more diverse and complex. As a result, banking business systems have become more diversified, and the correlation between systems has become more complex. Some complex business processes will go through several or even dozens of systems. When problems occur in such businesses, it may require multiple relevant business system managers to invest a lot of manpower and time to jointly investigate and finally confirm the problem. How to effectively simplify the problem discovery process and quickly locate the root cause of the problem has become a bottleneck for improving operation and maintenance efficiency, and is also one of the important issues that the current banking IT operation and maintenance center needs to solve.

[0003] At present, the operation and maintenance departments of the banking industry generally use the transaction full-link tracking system to realize the full life cycle link tracking function of a single business. The system obtains standardized transaction data of all transactions and stores them uniformly by collecting and analyzing the application logs of various business systems within the bank or the business traffic between systems in real time. Operation and maintenance personnel can search all transactions through certain key identifiers in the transaction full-link tracking system to identify a single problematic transaction. The transaction full-link tracking system uses the global unique identifier (transaction global serial number) in the standardized transaction message to perform a global search and transaction link assembly for the previous and subsequent transactions containing the same serial number, obtain the complete transaction link of the transaction and the processing status of the transaction in each application node in the transaction link, thereby realizing the ability to quickly locate the root cause of faults in transactions across multiple application systems.

[0004] However, there are three main current situations for small and medium-sized banks in link analysis based on transaction messages. First, due to historical reasons, many existing systems of banks have large differences in message formats; second, in order to ensure their own rapid development, many systems of small and medium-sized banks are introduced as products, and the systems are more heterogeneous; third, the global serial number specification is difficult to implement, and in reality, most transactions are difficult to use a unified serial number for correlation analysis. This makes it difficult to implement the global serial number specification in all business systems within the bank. For systems that perform full-link analysis of transactions based on global serial numbers, the transaction link analysis will be disconnected, which makes it difficult for operation and maintenance personnel to quickly locate faults. Summary of the invention

[0005] The purpose of the present invention is to provide a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry to solve the problems raised in the above-mentioned background technology.

[0006] The technical solution of the present invention is: a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry, comprising the following steps:

[0007] S1. Traffic collection: The real-time transaction message traffic between production servers is mirrored through the switch backplane mirroring function, and the mirrored traffic of multiple production network switches is filtered and aggregated through the network tap (TAP). The processed traffic is uniformly integrated into the system's business traffic pool.

[0008] S2. Traffic analysis: The system uses the collection task distributor to use the traffic packet mining process PktMiner in high speed and parallel to mine the required traffic packets on demand from the business traffic pool, and analyze and match the transaction traffic packets;

[0009] S3. Transaction concatenation: The full-link transaction analysis system for non-standardized transaction messages needs to combine conventional concatenation, expert model rule concatenation, and intelligent matching concatenation to gradually achieve comprehensive acquisition of the operation status of the entire transaction link and horizontally locate the root cause of the fault.

[0010] Furthermore, in S2, the traffic parsing includes parsing and matching, and the parsing includes: according to the message format between business systems, converting the binary network traffic bytes into formatted key, value pair message data; the matching includes matching the parsed business messages according to the connection method between business systems to obtain a complete business message.

[0011] Further, in S3, the conventional series connection includes: using multiple custom series connection identifiers for series connection.

[0012] Furthermore, in S3, the expert model rule concatenation includes: operation and maintenance personnel can concatenate models through custom business scenario full-link and upstream and downstream transaction matching rules.

[0013] Furthermore, in S3, the intelligent matching series connection includes: automatic series connection through time series + clustering + pattern recognition of transaction messages.

[0014] Furthermore, for a single transaction message, the optimal matching mode among the three matching modes of conventional concatenation, expert model rule concatenation, and intelligent matching concatenation is intelligently selected for message concatenation matching.

[0015] Furthermore, the step of intelligently selecting the optimal matching mode for message series matching includes:

[0016] S31. Use "conventional concatenation" to concatenate the initial message once to match the initial transaction link;

[0017] S32, judging whether the link generated by "conventional series connection" matches the expert model in the expert model library; if so, connecting the expert models in series;

[0018] S33. In the case of expert model mismatch, determine whether the link generated by "conventional series connection" has a circuit break; if there is a circuit break, it means that there must be non-standard messages in the link that do not contain the global serial number, and use intelligent matching series connection to perform automatic circuit break series connection matching to discover such non-standard messages.

[0019] Furthermore, for the intelligent matching and concatenation of non-standard messages, clustering and similarity matching algorithms are used to automatically match the downstream transaction messages with the highest similarity layer by layer starting from the breakpoint of the transaction link, until the matched messages are successfully spliced ​​to the downstream breakpoint and a complete link is formed, thereby achieving self-discovery of non-standard transaction paths in an unsupervised manner.

[0020] A full-link transaction analysis system based on non-standardized transaction messages in the banking industry, the full-link transaction analysis system comprising flow collection, flow analysis and transaction concatenation.

[0021] Furthermore, the transaction full-link analysis system uses the Jaccard similarity system to compare the similarity between upstream and downstream messages. For the upstream message A of the breakpoint, the system first filters out the message that meets the result set {B}. Each message {B1, B2...Bn} in the B set meets B.SRCIP=A.DSTIP and B.SedTime<=(A.RecTime+θ), where θ is the estimated average processing time within the system of the recipient of message A, B.SRCIP is the source address, A.DSTIP is the destination address, B.SedTime is the message sending time, and A.RecTime is the message receiving time.

[0022] For a given message A and the key / value set in each message Bn in the set B, the Jaccard similarity coefficient is defined as the ratio of the size of all intersections of A and Bn to the size of the union of A and Bn:

[0023]

[0024] The present invention provides a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry through improvement, which has the following improvements and advantages compared with the prior art:

[0025] Compared with the prior art, the present invention does not need to perform message standardization transformation on non-standard systems, thus saving the manpower and financial costs of system transformation and avoiding the business continuity risks that may be caused by system transformation; secondly, the Jaccard similarity coefficient is used to compare the similarity between upstream and downstream messages, and self-discovery of non-standard transaction paths is achieved in an unsupervised manner without human intervention; in addition, by combining conventional series connection, expert model rule series connection, and intelligent matching series connection, the expert model can be used for characteristic matching in scenarios where partial similarity matching cannot achieve satisfactory results, which is suitable for transaction series connection of most non-standardized messages. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments:

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is a structural block diagram of the transaction series connection of the present invention;

[0029] Figure 3 It is a structural block diagram of the intelligent matching of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The present invention provides a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry through improvement. The technical solution of the present invention is:

[0032] like Figure 1 As shown, a transaction full-link analysis method and system based on non-standardized transaction messages in the banking industry includes the following steps:

[0033] S1. Traffic collection: The real-time transaction message traffic between production servers is mirrored through the switch backplane mirroring function, and the mirrored traffic of multiple production network switches is filtered and aggregated through the network tap (TAP). The processed traffic is uniformly integrated into the system's business traffic pool.

[0034] S2. Traffic analysis: The system uses the collection task distributor to use the traffic packet mining process PktMiner in high speed and parallel to mine the required traffic packets from the business traffic pool on demand, and analyze and match the transaction traffic packets;

[0035] S3. Transaction concatenation: The full-link transaction analysis system for non-standardized transaction messages needs to combine conventional concatenation, expert model rule concatenation, and intelligent matching concatenation to gradually achieve comprehensive acquisition of the operation status of the entire transaction link and horizontally locate the root cause of the fault.

[0036] Among them, in S2, the traffic parsing includes parsing and matching, and the parsing includes: according to the message format between business systems (SOAP, fixed length, json, etc.), converting the binary network traffic bytes into formatted key, value pair message data; the matching includes matching the parsed business messages according to the connection mode between business systems (synchronous / asynchronous, long / short connection, etc.), so as to obtain a complete business message (request message + response message).

[0037] Among them, in S3, the conventional series connection includes: using multiple custom series connection identifiers (mainly global transaction serial numbers) for series connection.

[0038] Among them, in S3, the expert model rule series connection includes: the operation and maintenance personnel can connect the model through the custom business scenario full link and upstream and downstream transaction matching rules.

[0039] Among them, in S3, the intelligent matching series connection includes: automatic series connection through the method of time series + clustering + pattern recognition of transaction messages.

[0040] Among them, for a single transaction message, the optimal matching mode among the three matching modes of conventional concatenation, expert model rule concatenation, and intelligent matching concatenation is intelligently selected for message concatenation matching.

[0041] Among them, Figure 2 As shown, the step of intelligently selecting the optimal matching mode for message series matching includes:

[0042] S31. Use "conventional concatenation" to concatenate the initial message once to match the initial transaction link;

[0043] S32, judging whether the link generated by "conventional series connection" matches the expert model in the expert model library; if so, connecting the expert models in series;

[0044] S33. In the case of expert model mismatch, determine whether the link generated by "conventional series connection" has a circuit break; if there is a circuit break, it means that there must be non-standard messages in the link that do not contain the global serial number, and use intelligent matching series connection to perform automatic circuit break series connection matching to discover such non-standard messages.

[0045] Among them, Figure 3As shown, the intelligent matching and concatenation of non-standard messages uses clustering and similarity matching algorithms to automatically match the downstream transaction messages with the highest similarity layer by layer starting from the breakpoint of the transaction link until the matched messages are successfully spliced ​​to the downstream breakpoint and form a complete link, thereby realizing self-discovery of non-standard transaction paths in an unsupervised manner.

[0046] A full-link transaction analysis system based on non-standardized transaction messages in the banking industry, the full-link transaction analysis system comprising flow collection, flow analysis and transaction concatenation.

[0047] Among them, the transaction full-link analysis system uses the "Jaccard similarity coefficient" to compare the similarity between upstream and downstream messages. The system first parses and formats all single messages collected by bypass mode, and decomposes the message body into a set of formatted key-value pairs of key and value. For the upstream message A of the breakpoint, the system first screens out the result set {B} that meets the requirements. Each message {B1, B2...Bn} in the B set meets the requirements of B.SRCIP (source address) = A.DSTIP (destination address) and B.SedTime (message sending time) <= (A.RecTime (message receiving time) + θ), where θ is the estimated average processing time within the system of the A message receiver, B.SRCIP is the source address, A.DSTIP is the destination address, B.SedTime is the message sending time, and A.RecTime is the message receiving time.

[0048] For a given message A and the key / value set in each message Bn in the set B, the Jaccard similarity coefficient is defined as the ratio of the size of all intersections of A and Bn to the size of the union of A and Bn:

[0049]

[0050] The higher the Jaccard similarity coefficient of message B, the more likely it is to be the downstream message of message A. The system iterates in this cycle to automatically generate one or more possible closed-loop circuit-breaking paths.

[0051] Compared with the prior art, firstly, the present invention does not need to carry out message standardization transformation on the non-standard system, thus saving the manpower and financial costs of system transformation and avoiding the business continuity risks that may be caused by system transformation; secondly, the Jaccard similarity coefficient is used to compare the similarity between upstream and downstream messages, so as to realize the self-discovery of non-standard transaction paths in an unsupervised manner without human intervention; in addition, by combining conventional series connection, expert model rule series connection and intelligent matching series connection, the expert model can be used for characteristic matching in the scenario where some similarity matching cannot achieve satisfactory results, which is suitable for the transaction series connection of most non-standardized messages.

[0052] The above description enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A transaction full-link analysis method based on non-standardized transaction messages in the banking industry, characterized by: The following steps are involved: S1. Traffic collection: The real-time transaction message traffic between production servers is mirrored through the switch backplane mirroring function, and the mirrored traffic of multiple production network switches is filtered and aggregated through the network tap (TAP). The processed traffic is uniformly integrated into the system's business traffic pool. S2. Traffic analysis: The system uses the collection task distributor to use the traffic packet mining process PktMiner in high speed and parallel to mine the required traffic packets on demand from the business traffic pool, and analyze and match the transaction traffic packets; S3. Transaction concatenation: The full-link transaction analysis system for non-standardized transaction messages needs to combine conventional concatenation, expert model rule concatenation, and intelligent matching concatenation to gradually achieve comprehensive acquisition of the operation status of the entire transaction link and horizontally locate the root cause of the fault.

2. According to claim 1, a transaction full-link analysis method based on non-standardized transaction messages in the banking industry is characterized by: In S2, the traffic parsing includes parsing and matching. The parsing includes: converting binary network traffic bytes into formatted key, value pair message data according to the message format between business systems; the matching includes matching the parsed business messages according to the connection method between business systems to obtain a complete business message.

3. According to claim 1, a transaction full-link analysis method based on non-standardized transaction messages in the banking industry is characterized by: In S3, the conventional series connection includes: using multiple custom series connection identifiers for series connection.

4. According to claim 1, a transaction full-link analysis method based on non-standardized transaction messages in the banking industry is characterized by: In S3, the expert model rule concatenation includes: the operation and maintenance personnel can concatenate the model through the customized business scenario full link and upstream and downstream transaction matching rules.

5. According to claim 1, a transaction full-link analysis method based on non-standardized transaction messages in the banking industry is characterized by: In S3, the intelligent matching series connection includes: automatic series connection through time series + clustering + pattern recognition of transaction messages.

6. According to claim 1, a method for analyzing the entire transaction chain based on non-standardized transaction messages in the banking industry is characterized by: For a single transaction message, the optimal matching mode among the three matching modes of conventional concatenation, expert model rule concatenation and intelligent matching concatenation is intelligently selected for message concatenation matching.

7. A transaction full-link analysis method based on non-standardized transaction messages in the banking industry according to claim 6, characterized in that: The step of intelligently selecting the optimal matching mode for message series matching includes: S31. Use "conventional concatenation" to concatenate the initial message once to match the initial transaction link; S32, judging whether the link generated by "conventional series connection" matches the expert model in the expert model library; if so, connecting the expert models in series; S33. In the case where the expert model does not match, determine whether the link generated by the "conventional series connection" has a circuit break; if there is a circuit break, it means that there must be non-standard messages in the link that do not contain the global serial number, and use intelligent matching series connection to perform automatic circuit break series connection matching to discover such non-standard messages.

8. According to claim 1, a transaction full-link analysis method based on non-standardized transaction messages in the banking industry is characterized by: For the intelligent matching and concatenation of non-standard messages, clustering and similarity matching algorithms are used to automatically match the downstream transaction messages with the highest similarity layer by layer starting from the breakpoint of the transaction link until the matched messages are successfully spliced ​​to the downstream breakpoint and form a complete link, thus realizing self-discovery of non-standard transaction paths in an unsupervised manner.

Images