A transaction log string connection system and method based on interactive labeling and large model cooperation

Abstract

The application belongs to the technical field of transaction log analysis and big data processing, and discloses a transaction log concatenation system based on interactive labeling and large model cooperation, which comprises a configuration management module, a log retrieval and display module, an interactive labeling module, a pop-up window verification module, a rule generation module, a rule storage module, a rule backtesting module and a concatenation query module. The application adopts a page-based interactive labeling mode, can quickly complete candidate ID confirmation without professional coding, greatly reduces labor cost, gets rid of excessive dependence on the experience of analysts, solves the pain point of low efficiency of traditional manual labeling, simultaneously fuses feature method and large model method to cooperatively generate rules, takes into account the accuracy and business interpretability of concatenated rules, makes up for the defects of unreliable rules and poor accuracy of the pure automatic mode, and is matched with multi-view visual display and abnormal log highlighting to quickly locate fault nodes and reduce the cost of transaction link analysis and fault troubleshooting.

Description

Technical Field

[0001] This invention belongs to the field of transaction log analysis and big data processing technology, specifically a transaction log concatenation system and method based on interactive annotation and large model collaboration. Background Technology

[0002] In the banking transaction log business scenario, the amount of log data generated in a single day can reach tens of TB. The log of a complete transaction will be broken down and scattered across multiple business systems such as channels, ESB, and core. Moreover, the serial identifiers used by different systems do not have a unified standard in terms of naming rules, data format, and storage location, resulting in a fragmented distribution of transaction logs, which greatly hinders transaction link tracing and anomaly location.

[0003] Existing transaction log concatenation solutions have significant technical shortcomings: manual annotation methods heavily rely on the business experience of analysts, resulting in cumbersome procedures, low execution efficiency, and difficulty in quickly and stably generating reusable standardized concatenation rules; purely automated concatenation methods lack human intervention and verification, leading to poor interpretability of the generated rules and unreliable matching accuracy, increasing the overall cost of transaction concatenation, fault diagnosis, and end-to-end analysis. Therefore, there is an urgent need to develop a closed-loop solution that enables "rules to be mined, verified, reusable, and applicable" to meet the needs of banks for efficient analysis and operation and maintenance management of massive transaction logs. Summary of the Invention

[0004] The purpose of this invention is to provide a transaction log concatenation system and method based on interactive annotation and large model collaboration, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a transaction log concatenation system based on interactive annotation and large model collaboration, comprising a configuration management module, a log retrieval and display module, an interactive annotation module, a pop-up verification module, a rule generation module, a rule storage module, a rule backtesting module, and a concatenation query module. The configuration management module is used to maintain system metadata, ingress system, downstream system relationships and rule lists, and to define the concatenation analysis boundary and system topology path. The log retrieval and display module is used to retrieve transaction logs by time and management fields, and to visualize them in the form of a timeline + log text. The interactive annotation module is used to support manual selection of candidate IDs in the system or cross-system concatenation, and to highlight IDs selected in different modes differently. The pop-up verification module is used to display the filtering results, statistical data and cross-system verification information of the selected ID in a floating window to assist manual confirmation of the validity of the candidate ID; The rule generation module is used to automatically generate structured concatenated rules from manually confirmed candidate IDs based on rule feature method or large model method. The rule storage module is used to store complete rule data, including rule identifier ID, type, scope of application, extraction expression, time window, and management field constraints. The rule backtesting module is used to load historical logs to perform rule verification and output the statistical results of serialization hits and false serializations for manual review. The chain query module is used to execute cross-system chain queries based on effective rules, display the transaction chain in multiple views and highlight the abnormal logs.

[0006] Preferably, the system includes a rule management page, a concatenation exploration page, and a concatenation query page, supporting system addition, editing, and deletion, log data source index configuration, entry system marking, and downstream system list settings; the configuration management module corresponds to the rule management page, and the rule backtesting module is integrated into the rule management page; the log retrieval and display module, interactive annotation module, pop-up verification module, rule generation module, and rule storage module are all integrated into the concatenation exploration page; the concatenation query module corresponds to the concatenation query page.

[0007] Preferably, the interactive annotation module highlights single-system candidate IDs in yellow and cross-system candidate IDs in red, and only allows the selection of one candidate concatenation ID at a time.

[0008] Preferably, in a single-system verification scenario, the pop-up verification module displays the original log set on the left and the ID-filtered log set on the right, and synchronously outputs the number of hits, time span, and start and end time statistics. In a cross-system verification scenario, the pop-up verification module can switch to the downstream system and display the upstream system's filtering results, the downstream system's filtering results within the time window, and the relative time distribution of the upstream and downstream logs in the pop-up window.

[0009] Preferably, the rule feature method of the rule generation module is used to extract the character composition, length, prefix or suffix, separator, JSON key name or XML tag anchor features of the candidate ID, and automatically construct a regular expression extraction expression.

[0010] Preferably, the large model method of the rule generation module is used to input candidate ID samples, log context fragments, and system roles of a single system or across systems into a large model to generate accurate regular expression extraction expressions.

[0011] Preferably, the serial query module executes the serial query layer by layer according to the system topology, and must simultaneously satisfy rule hit, time window, and management field constraints. Multiple rules are combined using AND logic, and error logs are highlighted in red and warning logs are highlighted in yellow.

[0012] A method for concatenating transaction logs based on interactive annotation and large model collaboration, the specific steps of which are as follows: S1. Configure system metadata, entry system, and downstream system topology through the configuration management module to build the basic environment for serial analysis; S2. Use the log retrieval and display module to set the time range and manage field filtering conditions to retrieve and display the transaction logs of the target system; S3. Manually select candidate concatenation IDs through the interactive annotation module, trigger the pop-up verification module to display the verification results, and manually confirm the valid candidate IDs; S4. The rule generation module generates single-system / cross-system sequential rules based on the rule feature method or the large model method after confirming the candidate IDs. S5. The rule storage module stores rules in a structured manner. The storage fields include rule ID, type, applicable system, regular expression, time window, and management field constraints. S6. The rule backtesting module selects historical time period logs to perform rule backtesting, outputs the verification results, and allows manual review of rule availability. S7. The serial query module loads and applies the rules, executes cross-system serial queries according to the system topology, displays the complete transaction chain in multiple views and highlights the abnormal logs.

[0013] Preferably, in step S3, after the single-system verification is completed and manually confirmed, the single-system rule generation process begins; after the cross-system verification selects and confirms the downstream system, cross-system rules that mark the scope of the upstream system → downstream system are generated.

[0014] Preferably, in step S4, rule generation supports the combined use of feature-based methods and large-model methods. First, the feature-based method is used to extract basic anchor points to construct regularization, and then the large-model method is used to optimize the accuracy and adaptability of the regularization.

[0015] The beneficial effects of this invention are as follows: 1. This invention adopts a page-based interactive annotation mode, which can quickly complete the confirmation of candidate IDs without professional coding, greatly reducing labor costs and eliminating excessive reliance on the experience of analysts, thus solving the pain point of low efficiency in traditional manual annotation. At the same time, it integrates feature-based methods and large model methods to generate rules in a collaborative manner, taking into account both the accuracy of the concatenated rules and the business interpretability, making up for the defects of unreliable rules and poor accuracy in purely automated methods. With multi-view visualization and anomaly log highlighting, fault nodes can be quickly located, effectively reducing the cost of transaction link analysis and fault diagnosis.

[0016] 2. This invention constructs a closed-loop system integrating rule mining, verification, backtesting, and application, realizing the structured accumulation and reuse of serial rules. It also supports the flexible generation of rules for single systems or cross-systems, adapting to the fragmented transaction log scenarios of multiple systems in banks. It can efficiently support the serial analysis of massive transaction logs of tens of TB per day, fully meeting the core needs of banks for efficient operation and maintenance and transaction link traceability. Attached Figure Description

[0017] Figure 1 System module architecture diagram of this invention; Figure 2 Flowchart of the transaction log concatenation method of this invention; Figure 3 Schematic diagram of a single-system serial exploration page of the present invention; Figure 4 A schematic diagram of the cross-system serial exploration page of this invention; Figure 5 This invention presents a schematic diagram of the page displaying the results of a sequential query. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figures 1 to 5 As shown, this embodiment of the invention provides a transaction log concatenation system based on interactive annotation and large model collaboration, including a configuration management module, a log retrieval and display module, an interactive annotation module, a pop-up verification module, a rule generation module, a rule storage module, a rule backtesting module, and a concatenation query module. The configuration management module is used to maintain system metadata, entry system, downstream system relationships and rule lists, and define the boundary of the series analysis and the system topology path; The log retrieval and display module is used to retrieve transaction logs by time and management fields, and to visualize them in the form of a timeline + log text. The interactive annotation module is used to support manual selection of candidate IDs in the system or cross-system concatenation, and to highlight IDs selected in different modes differently. The pop-up verification module is used to display the filtering results, statistical data, and cross-system verification information of the selected ID in a floating window, assisting manual confirmation of the validity of the candidate ID; The rule generation module is used to automatically generate structured concatenation rules from manually confirmed candidate IDs based on rule feature method or large model method; The rule storage module is used to store complete rule data, including rule identifier ID, type, scope of application, extraction expression, time window, and management field constraints. The rule backtesting module is used to load historical logs to execute rule verification and output the statistical results of chain hits and false chaining for manual review; The chain query module is used to execute cross-system chain queries based on effective rules, display the transaction chain in multiple views and highlight the abnormal logs.

[0020] It adopts an eight-module design with clear division of labor and collaborative linkage, covering the entire process of transaction log connection, forming a complete closed loop of configuration, mining, confirmation, generation, backtesting, and application. Each module performs its own function, which can efficiently process tens of terabytes of massive logs from banks every day. It combines human interaction with large-scale model intelligence, eliminating the over-reliance on professional experience. It can ensure the interpretability and accuracy of rules, and realize the structured accumulation and reuse of rules. At the same time, through visualization and anomaly highlighting, it can quickly complete transaction link tracing and fault location, comprehensively improving log analysis efficiency and reducing operation and maintenance and analysis costs.

[0021] The system includes a rule management page, a concatenation exploration page, and a concatenation query page, supporting the addition, editing, and deletion of system entries, configuration of log data source indexes, marking of entry systems, and setting of downstream system lists. The configuration management module corresponds to the rule management page, and the rule backtesting module is integrated into the rule management page. The log retrieval and display module, interactive annotation module, pop-up verification module, rule generation module, and rule storage module are all integrated into the concatenation exploration page. The concatenation query module corresponds to the concatenation query page.

[0022] The precise correspondence between pages and modules, the aggregation and integration of functions, and the concise and clear operation path significantly reduce the barrier to entry and make it easy for business personnel to get started quickly. The integration of modules by page reduces switching costs, and the clear division of functions and responsibilities not only adapts to the bank's system configuration and rule management processes, but also facilitates system maintenance and function expansion, thereby improving overall operation and management efficiency.

[0023] The interactive annotation module highlights candidate IDs in yellow for single-system applications and in red for cross-system applications, and only one candidate ID can be selected at a time.

[0024] The candidate IDs are highlighted differently for single / cross-system applications, making it easy to distinguish operation scenarios and avoid confusion. Only one ID can be selected at a time, ensuring the accuracy of annotation and preventing multiple selections from interfering with verification and rule generation. This design simplifies manual operation, reduces the probability of misjudgment, and provides a reliable annotation foundation for subsequent rule generation.

[0025] In a single-system verification scenario, the pop-up verification module displays the original log set on the left and the ID-filtered log set on the right, simultaneously outputting the number of hits, time span, and start and end time statistics. In a cross-system verification scenario, the pop-up verification module can switch to the downstream system and display the upstream system's filtering results, the downstream system's filtering results within the time window, and the relative time distribution of upstream and downstream logs.

[0026] The system displays comparative data and statistical information for single / cross-system scenarios, intuitively presenting the ID aggregation effect and cross-system correlation, and can quickly determine the validity of candidate IDs; it supports switching downstream systems across systems, clearly showing the correlation between upstream and downstream logs, and assisting manual verification and confirmation in an efficient manner.

[0027] Among them, the rule feature method of the rule generation module is used to extract the character composition, length, prefix or suffix, delimiter, JSON key name or XML tag anchor features of candidate IDs, and automatically construct regular expression extraction expressions.

[0028] It automatically extracts multi-dimensional features of candidate IDs to construct regular expressions without manual coding, and is highly efficient, stable, and interpretable. It can adapt to various log formats such as delimiters, JSON, and XML, accurately match bank transaction log features, and quickly generate standardized concatenation extraction rules.

[0029] Among them, the large model method of the rule generation module is used to input candidate ID samples, log context fragments, and system roles of single system or cross-system into the large model to generate accurate regular expression extraction expressions.

[0030] By combining samples, context, and system roles, regular expressions can be intelligently generated, adapting to complex non-standard log formats and significantly improving the accuracy and adaptability of regular expressions; it also reduces the difficulty of manually writing rules and effectively solves the problem of the difficulty in automatically generating concatenated rules in complex scenarios.

[0031] The sequential query module executes sequentially layer by layer according to the system topology, and must simultaneously satisfy rule hit, time window, and management field constraints. Multiple rules are combined using AND logic, and error logs are highlighted in red and warning logs are highlighted in yellow.

[0032] The system is connected layer by layer according to topology, and multiple constraints are implemented using AND logic to ensure that the connection results are accurate and without redundancy; abnormal logs are highlighted in a hierarchical manner to quickly locate faulty nodes; it fully conforms to the transaction link analysis logic of multiple banking systems, significantly improving the efficiency of fault diagnosis and link tracing.

[0033] A method for concatenating transaction logs based on interactive annotation and large model collaboration, the specific steps of which are as follows: S1. Configure system metadata, entry system, and downstream system topology through the configuration management module to build the basic environment for serial analysis; S2. Use the log retrieval and display module to set the time range and manage field filtering conditions to retrieve and display the transaction logs of the target system; S3. Manually select candidate concatenation IDs through the interactive annotation module, trigger the pop-up verification module to display the verification results, and manually confirm the valid candidate IDs; S4. The rule generation module generates single-system / cross-system sequential rules based on the rule feature method or the large model method after confirming the candidate IDs. S5. The rule storage module stores rules in a structured manner. The storage fields include rule ID, type, applicable system, regular expression, time window, and management field constraints. S6. The rule backtesting module selects historical time period logs to perform rule backtesting, outputs the verification results, and allows manual review of rule availability. S7. The serial query module loads and applies the rules, executes cross-system serial queries according to the system topology, displays the complete transaction chain in multiple views and highlights the abnormal logs.

[0034] It adopts standardized procedures and processes to achieve a closed loop from system configuration to connected applications, making the operation standardized and easy to promote; the process integrates human interaction and intelligent generation, reducing reliance on professional experience; rules are generated, stored, backtested and verified before being put into use, ensuring the reliability and usability of the rules; it supports single / cross-system scenarios, adapts to massive log analysis, and multi-view visualization improves analysis efficiency, enabling rapid transaction link tracing and anomaly location, fully meeting the needs of bank log analysis.

[0035] In step S3, after the single-system verification is completed and manually confirmed, the single-system rule generation process begins; after the cross-system verification selects and confirms the downstream system, cross-system rules are generated that mark the scope of the upstream system to the downstream system.

[0036] Rules are generated by process segmentation based on single system and cross-system operations, accurately matching different application scenarios, clarifying the scope and applicable objects of the rules, avoiding rule confusion and misuse, improving the relevance and applicability of the rules, and perfectly adapting to the business needs of banks in single-system streamlining and cross-system link tracing.

[0037] In step S4, rule generation supports the combined use of feature-based and large-model methods. First, the feature-based method is used to extract basic anchor points to construct regularization, and then the large-model method is used to optimize the accuracy and adaptability of the regularization.

[0038] The feature-based approach ensures the reliability and interpretability of the rules, while the large model optimizes the accuracy and adaptability of regularization. The combination of these two approaches balances efficiency, accuracy, and interpretability. It can adapt to the complex and diverse transaction log formats of banks and generate optimal concatenation extraction rules.

[0039] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0040] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A transaction log concatenation system based on interactive annotation and large model collaboration, comprising a configuration management module, a log retrieval and display module, an interactive annotation module, a pop-up verification module, a rule generation module, a rule storage module, a rule backtesting module, and a concatenation query module, characterized in that: The configuration management module is used to maintain system metadata, ingress system, downstream system relationships and rule lists, and to define the concatenation analysis boundary and system topology path. The log retrieval and display module is used to retrieve transaction logs by time and management fields, and to visualize them in the form of a timeline + log text. The interactive annotation module is used to support manual selection of candidate IDs in the system or cross-system concatenation, and to highlight IDs selected in different modes differently. The pop-up verification module is used to display the filtering results, statistical data and cross-system verification information of the selected ID in a floating window to assist manual confirmation of the validity of the candidate ID; The rule generation module is used to automatically generate structured concatenated rules from manually confirmed candidate IDs based on rule feature method or large model method. The rule storage module is used to store complete rule data, including rule identifier ID, type, scope of application, extraction expression, time window, and management field constraints. The rule backtesting module is used to load historical logs to perform rule verification and output the statistical results of serialization hits and false serializations for manual review. The chain query module is used to execute cross-system chain queries based on effective rules, display the transaction chain in multiple views and highlight the abnormal logs.

2. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: The system includes a rule management page, a concatenation exploration page, and a concatenation query page, supporting the addition, editing, and deletion of system entries, configuration of log data source indexes, marking of entry systems, and setting of downstream system lists. The configuration management module corresponds to the rule management page, and the rule backtesting module is integrated into the rule management page. The log retrieval and display module, interactive annotation module, pop-up verification module, rule generation module, and rule storage module are all integrated into the concatenation exploration page. The concatenation query module corresponds to the concatenation query page.

3. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: The interactive annotation module highlights candidate IDs in yellow for single-system candidates and in red for cross-system candidate IDs, and only allows the selection of one candidate concatenation ID at a time.

4. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: In a single-system verification scenario, the pop-up verification module displays the original log set on the left and the ID-filtered log set on the right, simultaneously outputting statistics on the number of hits, time span, and start and end times. In a cross-system verification scenario, the pop-up verification module can switch to the downstream system and display the upstream system's filtering results, the downstream system's filtering results within the time window, and the relative time distribution of the upstream and downstream logs in the pop-up window.

5. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: The rule feature method of the rule generation module is used to extract the character composition, length, prefix or suffix, separator, JSON key name or XML tag anchor features of the candidate ID, and automatically construct regular expression extraction expressions.

6. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: The large model method of the rule generation module is used to input candidate ID samples, log context fragments, and system roles of a single system or cross-system into the large model to generate accurate regular expression extraction expressions.

7. The transaction log string association system based on interactive labeling and large model cooperation according to claim 1, characterized in that: The sequential query module executes sequentially layer by layer according to the system topology, and must simultaneously satisfy rule hit, time window, and management field constraints. Multiple rules are combined using AND logic, and error logs are highlighted in red and warning logs are highlighted in yellow.

8. A method for transaction log concatenation based on interactive labeling and large model cooperation, characterized in that, The specific steps are as follows: S1. Configure system metadata, entry system, and downstream system topology through the configuration management module to build the basic environment for serial analysis; S2. Use the log retrieval and display module to set the time range and manage field filtering conditions to retrieve and display the transaction logs of the target system; S3. Manually select candidate concatenation IDs through the interactive annotation module, trigger the pop-up verification module to display the verification results, and manually confirm the valid candidate IDs; S4. The rule generation module generates single-system / cross-system sequential rules based on the rule feature method or the large model method after confirming the candidate IDs. S5. The rule storage module stores rules in a structured manner. The storage fields include rule ID, type, applicable system, regular expression, time window, and management field constraints. S6. The rule backtesting module selects historical time period logs to perform rule backtesting, outputs the verification results, and allows manual review of rule availability. S7. The serial query module loads and applies the rules, executes cross-system serial queries according to the system topology, displays the complete transaction chain in multiple views and highlights the abnormal logs.

9. The method for transaction log concatenation based on interactive labeling and large model cooperation according to claim 8, characterized in that: In step S3, after the single-system verification is completed and manually confirmed, the single-system rule generation process begins; after the cross-system verification selects and confirms the downstream system, cross-system rules are generated that mark the scope of the upstream system → downstream system.

10. The method for concatenating transaction logs based on interactive annotation and large model collaboration according to claim 8, characterized in that: In step S4, rule generation supports the combined use of feature-based and large-model methods. First, the feature-based method is used to extract basic anchor points to construct regularization, and then the large-model method is used to optimize the accuracy and adaptability of the regularization.

Images