A service log index data cleaning method, device and equipment and storage medium

Abstract

The application discloses a kind of service log index data cleaning method, device, equipment and storage medium.It includes: obtaining original service log, original service log is parsed, generates key transaction field;Key transaction field is mapped to fixed field, and standard field is generated;Business index is calculated based on standard field, and business index and original log are stored based on double-layer storage architecture.Through generating key transaction field, the unified analysis of different format logs can be realized, the success rate and consistency of extraction are guaranteed, and non-predefined field rich data dimension can also be intelligently mined.Through fixed field mapping, the naming and format of each system field can be unified, and a global standardized field system is established.Through double-layer storage architecture, index and original log can be stored, unified intelligent calculation of business index can be realized, real and accurate index is guaranteed, differentiated storage management of data can also be realized, long-term retention and cost optimization are considered, and data access efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, device, and storage medium for cleaning business log indicator data. Background Technology

[0002] In the process of digital transformation in the banking, insurance, securities and other financial industries, institutions within the industry have accumulated a large number of business systems covering multiple business areas over a long period of time. The stable and efficient operation of these business systems is the core foundation for ensuring the normal conduct of financial transactions and the orderly advancement of industry business. Therefore, implementing comprehensive and effective transaction monitoring of business systems has become a key task for the technical operation and maintenance and business support of the financial industry.

[0003] Currently, the monitoring of various business systems in the financial industry generally adopts an independent management model for each system. That is, each business system either has its own supporting monitoring functions or does not have a dedicated monitoring mechanism. There is no unified monitoring system covering all business systems in the industry, nor is there a standardized business log processing procedure.

[0004] Current monitoring models lack a unified management system, making it impossible to distribute consistent monitoring strategies to all business systems and achieve centralized transaction monitoring and management. Furthermore, the log data generated by various business systems is not standardized in format and field type, requiring separate development of adaptation programs for each system's log data, resulting in significant manpower costs. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for cleaning business log indicator data. By constructing a standardized business log access, parsing, and mapping system, as well as an intelligent indicator calculation, storage, and monitoring system, it solves the technical problems of fragmented business system monitoring, non-standard data formats, the need for separate development and adaptation, and high manpower costs.

[0006] According to one aspect of the present invention, a method for cleaning business log indicator data is provided, the method comprising: Obtain the original business logs, parse the original business logs, and generate key transaction fields; Perform fixed field mapping on key transaction fields to generate standard fields; Business metrics are calculated based on standard fields, and business metrics and raw logs are stored using a two-tier storage architecture.

[0007] Optionally, the original business logs can be obtained, including: obtaining a standardized template; and configuring the system to be cleaned to access the system using the standardized template in order to obtain the original business logs.

[0008] Optionally, the original business logs are parsed to generate key transaction fields, including: determining the format type of the original business logs and determining the preset priority corresponding to each format type; parsing the original business logs according to the preset priority to extract key transaction fields, wherein the key transaction fields include a unique serial number, a global serial number, a transaction status, and a transaction duration.

[0009] Optionally, a fixed field mapping is performed on the key transaction fields to generate standard fields, including: obtaining a fixed field template, wherein the fixed field template includes each fixed field; determining the target field corresponding to the key transaction field from the fixed field template; and performing a fixed field mapping on the key transaction fields based on the target field to generate each standard field.

[0010] Optional methods also include: performing a full parsing test based on standard fields and calculating the parsing success rate; triggering an alarm when the parsing success rate is lower than a preset threshold.

[0011] Optionally, business metrics can be calculated based on standard fields, including: extracting, validating and calculating data for each standard field according to a preset period to generate various business metrics, including transaction volume, transaction time, transaction success rate, transaction response rate and error log rate.

[0012] Optionally, business metrics and raw logs are stored based on a two-tier storage architecture, including: establishing a metric index and a tracking index; storing business metrics in the metric index and setting a first specified period, and storing raw logs in the tracking index and setting a second specified period, wherein the first specified period is longer than the second specified period.

[0013] According to another aspect of the present invention, a business log indicator data cleaning apparatus is provided, the apparatus comprising: The key transaction field generation module is used to obtain raw business logs, parse the raw business logs, and generate key transaction fields; The standard field generation module is used to perform fixed field mapping on key transaction fields and generate standard fields. The metrics and log storage module is used to calculate business metrics based on standard fields and to store business metrics and raw logs based on a two-tier storage architecture.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform a business log indicator data cleaning method according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement a business log indicator data cleaning method according to any embodiment of the present invention.

[0016] The technical solution of this invention, by generating key transaction fields, enables unified parsing of logs in different formats, ensuring the success rate and consistency of extraction, and intelligently mining non-predefined fields to enrich data dimensions. Through fixed field mapping, the naming and format of fields across systems can be unified, establishing a globally standardized field system. By storing metrics and raw logs through a two-layer storage architecture, unified intelligent calculation of business metrics can be achieved, ensuring the accuracy and reliability of metrics. It also enables differentiated data storage management, balancing long-term retention and cost optimization, and improving data access efficiency.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a business log indicator data cleaning method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of another business log indicator data cleaning method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a business log indicator data cleaning device provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements a business log indicator data cleaning method according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1 Figure 1 This is a flowchart illustrating a business log indicator data cleaning method according to Embodiment 1 of the present invention. This embodiment is applicable to business log processing scenarios. The method can be executed by a business log indicator data cleaning device, which can be implemented in hardware and / or software and can be configured in a computer controller. Figure 1 As shown, the method includes: S110. Obtain the original business logs, parse the original business logs, and generate key transaction fields.

[0023] Among them, raw business logs refer to the native log data generated by the business system during operation, which are the original records of business transactions. Key transaction fields refer to the fields extracted from the raw business logs that can characterize the core features of business transactions, and are the core identifiers and key parameters of transaction behavior.

[0024] Optionally, the original business logs can be obtained, including: obtaining a standardized template; and configuring the system to be cleaned to access the system using the standardized template in order to obtain the original business logs.

[0025] Specifically, in the process of obtaining raw business logs, obtaining a standardized template refers to first developing a standardized CSV template adapted to various business systems in the financial industry. This standardized template serves as a unified system access configuration carrier, pre-setting core configuration items required for the access of the business systems to be cleaned. These include basic information items such as the system's Chinese name, application's English identifier, and log type tags. It also specifies the specific format of the log source, covering various mainstream log format types such as JSON, XML, key-value pairs, and plain text. A sample log submission item is also provided to support subsequent rule verification. Configuring access to obtain raw business logs through the standardized template involves using the standardized CSV template as a unified entry point to configure access for various business systems requiring log cleaning. Operators must fill in the actual information of each system to be cleaned according to the template's pre-set configuration items, specifying the system's Chinese name, application's English identifier, log type tags, selecting the specific format of the system's log source, and simultaneously providing a sample log from that system. After completing the access configuration for a single system, the configuration information is imported into the log cleaning system. All business systems to be cleaned are accessed according to the configuration requirements of this standardized template. After receiving configuration information from each system, the log cleaning system will pull the corresponding raw log data generated during operation from each business system to be cleaned based on the log source format and system information specified in the configuration. This achieves unified acquisition of raw business logs. At the same time, all imported configuration information will be version-managed to ensure that any configuration changes are auditable. Tests will be automatically triggered during import to verify the feasibility of subsequent parsing rules using submitted sample logs. This ensures accurate configuration and prepares for the subsequent parsing of raw business logs, allowing raw business logs of different formats and sources to be uniformly aggregated into the log cleaning system through a standardized configuration process.

[0026] Optionally, the original business logs are parsed to generate key transaction fields, including: determining the format type of the original business logs and determining the preset priority corresponding to each format type; parsing the original business logs according to the preset priority to extract key transaction fields, wherein the key transaction fields include a unique serial number, a global serial number, a transaction status, and a transaction duration.

[0027] Specifically, the system will determine the format type of the original business logs and the preset priority of each format type. After the system completes the standardized CSV template access configuration of the business systems to be cleaned, the system will extract the specific format type of the original business log declaration of each system from the configuration information. The format types mainly include JSON, XML, key-value pairs, plain text, regular expression pattern recognition, etc. At the same time, the system will pre-set the parsing priority of the log format. The priority order is to perform JSON / XML parsing first, then key-value pair parsing, and finally regular expression pattern recognition parsing.

[0028] Furthermore, the system will activate the field parsing engine. Based on the log format type declared in the configuration of each system to be cleaned, it will automatically route the log parsing task to the most suitable parser, and then parse and process the original business logs according to the preset parsing priority. During the parsing process, the engine will enforce field extraction operations through predefined precise rules such as JSON paths and specific regular expression patterns, focusing on extracting key transaction fields that can represent the core characteristics of the transaction, specifically including unique serial numbers, global serial numbers, transaction status, and transaction time. At the same time, based on the extraction of core key transaction fields, the field parsing engine will also perform dynamic pattern analysis on the original business logs, intelligently discovering and extracting other non-predefined field information. While ensuring the complete extraction of core key transaction fields, it enriches the data dimensions and provides more comprehensive data support for subsequent field mapping and indicator calculation.

[0029] S120. Perform fixed field mapping on key transaction fields to generate standard fields.

[0030] Fixed field mapping refers to the process of converting non-standardized original field names extracted from different system logs into globally unique standard fixed field names according to a preset unified field template. Standard fields are transaction fields obtained after fixed field mapping that follow a globally unified naming convention and represent the final result of cross-system data standardization.

[0031] S130: Calculate business metrics based on standard fields, and store business metrics and raw logs based on a two-layer storage architecture.

[0032] Among them, business metrics refer to quantitative indicators that reflect the operational status and transaction performance of a business system, calculated based on standard fields and through preset calculation logic, intelligent rules, and dynamic strategies. The dual-layer storage architecture is a layered storage system designed to achieve optimal cost, efficient access, and full lifecycle governance for business metrics and raw logs. It includes two index layers: the metric index (Metric) and the tracking index (Trace_message).

[0033] Optionally, business metrics can be calculated based on standard fields, including: extracting, validating and calculating data for each standard field according to a preset period to generate various business metrics, including transaction volume, transaction time, transaction success rate, transaction response rate and error log rate.

[0034] The preset period can be 1 minute. Data extraction refers to the process by which the indicator calculation engine retrieves specific standard field data required for calculating each business indicator from a standardized field stream within each preset period. Data validation refers to the process of verifying the validity and completeness of the extracted standard field data before indicator calculation. Simultaneously, it uses built-in system rules to filter invalid data, such as automatically identifying and filtering test traffic and invalid global transaction IDs when calculating transaction volume, ensuring the authenticity and validity of the data used in the calculation and guaranteeing the accuracy of the final business indicators. Transaction volume is an indicator reflecting the scale of transactions in the business system, reflecting the actual total number of business transactions within the period. Transaction time reflects the transaction processing efficiency of the business system.

[0035] Specifically, the metric calculation engine uses pre-set intelligent rules and dynamic strategies to process standard field data in a targeted manner, ensuring the accuracy and authenticity of each business metric calculation. When calculating transaction volume, the engine extracts global transaction ID field data from the standard fields on a 1-minute cycle. It first counts the number of unique global transaction ID values, and automatically identifies and filters test traffic and invalid global transaction IDs to eliminate interference from non-real business data, ultimately obtaining the real business transaction volume data for that cycle. When calculating transaction latency metrics, the engine also extracts latency field data from the standard fields on a 1-minute cycle, prioritizing the direct acquisition of the field value as the latency of a single transaction. If the latency field is missing, the engine automatically activates the transaction link tracing mode, using the transaction flow ID or global transaction ID as the association key, and associates all log information of the same transaction within the corresponding time window, including logs of each stage such as request, processing, and response. By calculating the difference between the maximum and minimum timestamps in these logs, the engine dynamically obtains the precise latency of the transaction, and then completes the overall statistical calculation of transaction latency for that cycle. When calculating the transaction success rate, the transaction return code field data is extracted from the standard fields in a 1-minute cycle. Transactions with return codes of 00000 and 99999 are considered successful, while all other return codes are considered failed. The transaction success rate is obtained by statistically analyzing the ratio of successful transactions to the total number of transactions. The engine also supports a strategy learning function, which records high-frequency successful return codes that appear under different transaction service IDs and prompts operators to include these high-frequency successful codes in the custom success rules of the corresponding service, continuously optimizing the transaction success rate calculation strategy. When calculating the transaction response rate, data from the initiator and requester identification fields in the standard fields are extracted every 1 minute. First, logs explicitly marked with both the initiator and the responder are statistically analyzed. Once it's confirmed that both parties' logs for the same transaction exist, they are included in the valid statistics. The base response rate is calculated as the ratio of the valid statistical count to the total number of transactions. For logs without explicitly marked initiator and responder, the engine initiates sequence pattern analysis, aggregating logs based on the transaction transaction ID. By identifying the occurrence of transaction return information and specific keywords, it intelligently determines the logical relationship between multiple logs under the same transaction ID, inferring whether they constitute a complete request-response loop. Transactions determined to be closed loops are included in the valid statistics, and the final transaction response rate for that period is calculated comprehensively. When calculating the error log rate, data from the log level field in the standard fields is extracted every 1 minute. The number of error logs with a log level of "error" within that period is counted. The error log rate is directly calculated as the ratio of the number of error logs to the total number of logs within that period.

[0036] Optionally, business metrics and raw logs are stored based on a two-tier storage architecture, including: establishing a metric index and a tracking index; storing business metrics in the metric index and setting a first specified period, and storing raw logs in the tracking index and setting a second specified period, wherein the first specified period is longer than the second specified period.

[0037] The dual-tier storage architecture is a layered storage system designed for intelligent governance of business metrics and raw log data throughout their entire lifecycle. It divides storage into different tiers based on data value density, access patterns, and business requirements. Each tier has independent storage content, a retention period, and optimization strategies, balancing optimal data storage cost, efficient access, and long-term retention needs. The metric index is one of the storage indexes in the dual-tier architecture, specifically used to store processed and aggregated business metric data. It serves as the dedicated storage medium for business metrics, enabling efficient management and retrieval of time-series business metric data. The tracking index is the other storage index in the dual-tier architecture, primarily used to store raw or lightly enriched standardized detailed log data. It serves as the dedicated storage medium for raw business logs, preserving complete log context information and supporting log tracing and in-depth analysis. The first specified period is the data storage period set for the metric index, i.e., the retention time of business metric data in the metric index; the first specified period can be 360 ​​days. The second specified period is the dedicated data storage period set for the tracking index, i.e., the retention time of raw log data in the tracking index; the second specified period can be 180 days.

[0038] Specifically, the system establishes a metric index and a tracking index. These two indexes allow for the separate storage of log-related data of different types and values, facilitating efficient subsequent querying and management. Then, the system stores business metrics in the metric index and sets a first specified period. All business metrics calculated by the metric calculation engine, including transaction volume, transaction time, transaction success rate, transaction response rate, and error log rate, are stored in the pre-built metric index, with a first specified period of 360 days. This means the business metric data is retained in the metric index for 360 days. Furthermore, the metric index is equipped with an adaptive downsampling intelligent storage strategy. Original 1-minute precision business metric data is automatically aggregated to 5-minute precision after 30 days, to 1-hour precision after 90 days, and to daily precision after 180 days. Then, the system stores the raw logs in the tracking index and sets a second specified period. All raw business logs from the business system, after standardization and slightly enriched detail logs, are stored in the tracking index, with a second specified period of 180 days. This means that the raw log-related data is retained in the tracking index for 180 days, shorter than the first specified period of the metric index. This is because the raw log data is massive and has relatively low value density, so long-term retention is unnecessary. This balances the business needs of log tracing and problem investigation with the rational use of storage resources. The tracking index also comes with a dedicated storage optimization strategy. The system automatically analyzes query patterns and creates inverted indexes for high-frequency query fields such as logid and retcode, and columnar storage for range query fields such as timestamp and cost. This allows complex queries on raw logs to achieve millisecond-level response times, improving data access efficiency. Overall, by establishing two dedicated indexes and setting one long and one short storage period for them, business metrics and raw logs can be classified and stored in a two-tier storage architecture. This satisfies the different business needs of long-term analysis of business metrics and short-term tracing of raw logs, while achieving optimal control of storage costs through intelligent optimization strategies for each index, thus completing intelligent governance of the entire lifecycle of log-related data.

[0039] The technical solution of this invention, by generating key transaction fields, enables unified parsing of logs in different formats, ensuring the success rate and consistency of extraction, and intelligently mining non-predefined fields to enrich data dimensions. Through fixed field mapping, the naming and format of fields across systems can be unified, establishing a globally standardized field system. By storing metrics and raw logs through a two-layer storage architecture, unified intelligent calculation of business metrics can be achieved, ensuring the accuracy and reliability of metrics. It also enables differentiated data storage management, balancing long-term retention and cost optimization, and improving data access efficiency.

[0040] Example 2 Figure 2This is a flowchart of a business log indicator data cleaning method provided in Embodiment 2 of the present invention. This embodiment adds a specific process for generating standard fields by mapping key transaction fields to fixed fields, based on Embodiment 1. The specific content of steps S250-S260 is largely the same as steps S120-S130 in Embodiment 1, and therefore will not be repeated in this embodiment. Figure 2 As shown, the method includes: S210. Obtain the original business logs, parse the original business logs, and generate key transaction fields.

[0041] Optionally, the original business logs can be obtained, including: obtaining a standardized template; and configuring the system to be cleaned to access the system using the standardized template in order to obtain the original business logs.

[0042] Optionally, the original business logs are parsed to generate key transaction fields, including: determining the format type of the original business logs and determining the preset priority corresponding to each format type; parsing the original business logs according to the preset priority to extract key transaction fields, wherein the key transaction fields include a unique serial number, a global serial number, a transaction status, and a transaction duration.

[0043] S220. Obtain the fixed field template, which includes each fixed field.

[0044] The fixed field template is a pre-defined, unified field specification template designed to standardize log fields across various business systems in the financial industry. It serves as the basis for field mapping. The template includes all the preset fixed fields required for business log indicator calculation and transaction monitoring, covering transaction-related fields such as timestamp, log level, transaction ID, and global transaction ID. It also includes extended fields such as initiator / requester identifier, specific system name, and spare fields. All fields use a globally unified fixed naming format.

[0045] Specifically, the fixed field template is a unified field specification template pre-defined by the system to achieve cross-business system field standardization. The template contains all the fixed fields set for financial business log monitoring and analysis, including fields such as timestamp, log level, transaction log (IDLogid), global transaction ID (Globalid), transaction service ID (Service), time spent (Cost), transaction return code (Retcode), transaction return information (Retmsg), as well as fields such as initiator or requester (Type), specific system name (Typesub), and spare fields.

[0046] S230. Determine the target field corresponding to the key transaction field from the fixed field template.

[0047] Among them, the target field refers to the corresponding fixed field that is accurately matched to each key transaction field from the fixed field template according to the principle of consistency between the field's business attributes and actual meaning. It is a standardized name carrier after the key transaction field is mapped. For example, the extracted "unique serial number" is matched with the transaction serial number IDLogid in the template, and the "transaction time" is matched with the time cost in the template. Each key transaction field has a corresponding exclusive target field.

[0048] Specifically, the system first sorts out all key transaction fields extracted from the original logs of various business systems, including fields such as unique serial number, global serial number, transaction status, and transaction time. Then, according to the matching principle of business attributes and field meanings, it accurately matches the corresponding fixed field as the target field in the fixed field template for each key transaction field. For example, the extracted unique serial number is matched with the transaction serial number in the template, the global serial number is matched with the global transaction, and the transaction time is matched with the time consumption. This ensures that each key transaction field can find a corresponding target field with the same meaning in the fixed field template, achieving accurate correspondence between non-standardized fields and standardized fixed fields.

[0049] S240. Based on the target field, perform fixed field mapping on the key transaction fields to generate various standard fields.

[0050] Fixed field mapping refers to the process of converting key transaction fields extracted from different business systems, which have inconsistent naming and format, into target field names and standardized forms in a fixed field template according to the correspondence between key transaction fields and target fields. It is the core link in achieving field standardization in various business systems. After mapping, similar transaction fields in all systems adopt globally unified naming and forms.

[0051] Specifically, the system will, according to the established correspondence between key transaction fields and target fields, uniformly convert key transaction fields extracted from different business systems, which have inconsistent names and formats, into corresponding target field names in a fixed field template, thus completing the standardized mapping of fields. During the mapping process, the system will perform strict standardization processing on the mapping results to ensure that regardless of the original naming and format of the key transaction fields, they will all be uniformly named and formatted in the fixed field template after mapping, ultimately generating standard fields that conform to globally unified naming conventions.

[0052] Optional methods also include: performing a full parsing test based on standard fields and calculating the parsing success rate; triggering an alarm when the parsing success rate is lower than a preset threshold.

[0053] Specifically, the system automatically selects the 1,000 most recent transaction logs as test samples. The number of samples can be flexibly configured according to actual business needs. For these sample logs, a full parsing process will be performed according to the configured parsing rules and field mapping strategies. The entire process is based on the standard fields generated after mapping. Not only will the overall parsing success rate be calculated, but the parsing success rate of each standard field will also be calculated independently. This verifies the feasibility of the overall parsing rules and accurately locates the parsing problems of individual standard fields. For example, the independent parsing status of core standard fields such as timestamp, transaction ID, and global transaction ID will be calculated and statistically analyzed separately. The system will display the overall parsing success rate and the independent parsing success rate of each standard field in real time, allowing relevant personnel to intuitively grasp the parsing quality.

[0054] Furthermore, the system pre-sets a 90% success rate threshold for parsing. When the overall parsing success rate falls below 90%, or the independent parsing success rate of any core standard field falls below 90%, an alarm mechanism is automatically triggered to promptly notify relevant technical and operations personnel for intervention. Simultaneously, the system provides comprehensive support functions for handling issues after alarms. Relevant personnel can intuitively view the original logs of parsing failures, the extracted field results, and field mapping relationships within the system. It also supports online direct correction of parsing rules or adjustment of field mapping configurations, forming a closed-loop process from monitoring to discovering parsing problems, triggering alarms, manual intervention, and configuration optimization. Through continuous optimization of parsing rules and mapping configurations, the parsing success rate is continuously improved, ensuring the data quality of standard fields and guaranteeing that subsequent business metrics calculated based on standard fields accurately reflect the operational status of the business system.

[0055] S250 calculates business metrics based on standard fields and stores business metrics and raw logs based on a two-tier storage architecture.

[0056] Optionally, business metrics can be calculated based on standard fields, including: extracting, validating and calculating data for each standard field according to a preset period to generate various business metrics, including transaction volume, transaction time, transaction success rate, transaction response rate and error log rate.

[0057] Optionally, business metrics and raw logs are stored based on a two-tier storage architecture, including: establishing a metric index and a tracking index; storing business metrics in the metric index and setting a first specified period, and storing raw logs in the tracking index and setting a second specified period, wherein the first specified period is longer than the second specified period.

[0058] Optional methods also include: building standardized analysis dashboards based on stored business metrics to achieve dashboard display of key metrics, metric trend analysis and comparison, visualization of multi-dimensional service topology diagrams, in-depth service performance analysis, display of time consumption decomposition flame diagrams, and error pattern clustering analysis.

[0059] Specifically, the implementation of the Gold Index Dashboard involves displaying core business metrics across the entire business system in real-time within the analysis dashboard. These metrics include trading volume, transaction success rate, average transaction duration, and error log rate. The dashboard also uses a traffic light-like red, yellow, and green indicator to visually represent the system's health status for each metric, allowing relevant personnel to quickly grasp the overall system operation and achieve real-time monitoring of core indicators. Furthermore, the implementation of indicator trend analysis and comparison allows the analysis dashboard to retrieve various core business indicator data within any time range, automatically generating indicator trend charts. Based on stored historical indicator data, it also provides year-on-year and month-on-month comparisons for the indicators. The system performs comparative analysis and intelligent comparison with preset baselines, automatically identifying and marking abnormal fluctuations in indicator trends to help analysts discover abnormal patterns in indicators and uncover potential problems in system operation. It also enables multi-dimensional service topology visualization, building a multi-dimensional service topology diagram in the form of a service dependency graph within the stored business indicators, such as traffic and error rates related to each transaction service ID. This visualizes the call relationships between service modules and integrates key indicators such as transaction volume and error log rate into the nodes of the topology diagram, allowing operators to drill down into any service node for rapid root cause analysis and clarification of service calls. The analysis addresses problematic links in the supply chain; it enables in-depth service performance analysis, catering to the professional analysis needs of development and operations teams. In the analysis dashboard, stored business metric data is aggregated by transaction service ID, displaying the P50, P95, and P99 percentiles of transaction latency for each service module, as well as error log rates and service call volume rankings. This quantified metric ranking quickly identifies performance bottleneck modules and high-risk services in the system, providing precise directions for performance optimization. Furthermore, it implements a flame graph display of latency decomposition, further refining the stored transaction latency metric data for specific service modules within the analysis dashboard. This decomposes the overall transaction latency into database query time, external interface call time, and other parameters. The time consumption of specific sub-dimensions such as time is presented visually in the form of a flame graph, showing the distribution of time consumption for each sub-dimension. This allows for precise identification of time bottlenecks at the code level or in external dependencies, facilitating targeted performance optimization. Error pattern clustering analysis is implemented by retrieving transaction return codes and related data from stored business metrics in the analysis dashboard. Semantic clustering algorithms are used to intelligently categorize massive amounts of transaction return codes and information, summarizing complex error information into a limited number of typical error patterns. Simultaneously, the frequency of occurrence and the scope of business impact of each error pattern are statistically analyzed, helping relevant personnel prioritize errors and guiding technical personnel to carry out error repair work according to priority, thereby improving the efficiency of problem solving.

[0060] Optional methods also include: building an intelligent transaction monitoring and prediction system based on stored business metrics, establishing dynamic baselines for each service and core metric through automatic system learning, triggering dynamic baseline alarms when real-time metrics continuously deviate from the dynamic baselines; and using machine learning models to analyze the correlation between various business metrics, automatically identifying and alarming on hidden abnormal patterns of multi-metric correlations.

[0061] Specifically, the system retrieves historical business indicator data stored in the indicator index and selects indicator data from the same period in the past as learning samples. The system automatically learns and analyzes its historical operating patterns to calculate the fluctuation range of the indicator during normal operation, establishing a dynamic baseline that includes the upper and lower boundaries of normal fluctuations. This baseline conforms to the actual operating characteristics of the indicator, rather than being a fixed static value. During real-time system operation, it continuously collects real-time data from various services and core indicators and compares the real-time indicator data with the corresponding dynamic baseline. When the real-time indicator data does not exceed the baseline range only once, but continuously deviates from the normal fluctuation boundary of the dynamic baseline, the system automatically triggers a dynamic baseline alarm, promptly notifying relevant operations and maintenance personnel to intervene and investigate. This alarm method based on dynamic baselines effectively adapts to the normal fluctuation patterns of business indicators, reducing false alarms caused by normal indicator fluctuations and improving the accuracy of alarms.

[0062] Secondly, the system inputs various stored business indicator data into machine learning models such as Isolation Forest and Clustering. These models learn and uncover the inherent relationships between various business indicators, including transaction volume, transaction time, transaction success rate, and error log rate. They grasp the normal correlation patterns between these indicators, such as a slight increase in transaction time when transaction volume rises, or an inverse correlation between error log rate and transaction success rate. During real-time monitoring, the model continuously analyzes the correlation changes of various real-time business indicators. When it identifies hidden abnormal patterns that deviate from normal correlation patterns—for example, a sudden drop in transaction volume but a unchanged transaction success rate, a significant surge in transaction time but no corresponding increase in error log rate, or a decrease in transaction response rate but no significant change in transaction volume—the machine learning model automatically identifies these anomalies and triggers corresponding alarm mechanisms. This alerts relevant personnel to promptly analyze and investigate potential system problems. Through the analysis of multi-indicator correlations, comprehensive monitoring of the business system's operational status is achieved. This enables the discovery of complex anomalies that traditional monitoring methods cannot cover, proactively mitigating system operational risks and ensuring the stable operation of various business systems in the financial industry.

[0063] The technical solution of this invention, by mapping key transaction fields to fixed fields and generating standard fields, and by obtaining preset fixed field templates, matching corresponding target fields for key transaction fields and carrying out standardized mapping, can establish a globally unified field naming and standardization system, realize the unified conversion of non-standard fields in different business systems, solve the problem of inconsistent data formats in various systems, and provide a standardized field data foundation for subsequent unified calculation of business indicators and cross-system data interoperability.

[0064] Example 3 Figure 3 This is a schematic diagram of a business log indicator data cleaning device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a key transaction field generation module 310, used to acquire raw business logs, parse the raw business logs, and generate key transaction fields; The standard field generation module 320 is used to perform fixed field mapping on key transaction fields and generate standard fields. The metrics and log storage module 330 is used to calculate business metrics based on standard fields and store business metrics and raw logs based on a two-layer storage architecture.

[0065] Optionally, the key transaction field generation module 310 specifically includes: a raw log acquisition unit, used to: acquire a standardized template; and configure the access to the business system to be cleaned through the standardized template to acquire raw business logs.

[0066] Optionally, the key transaction field generation module 310 specifically includes: a key transaction field parsing unit, used to: determine the format type of the original business log and determine the preset priority corresponding to each format type; parse the original business log according to the preset priority to extract key transaction fields, wherein the key transaction fields include a unique serial number, a global serial number, a transaction status, and a transaction duration.

[0067] Optionally, the standard field generation module 320 is specifically used for: obtaining a fixed field template, wherein the fixed field template includes various fixed fields; determining the target field corresponding to the key transaction field from the fixed field template; and performing fixed field mapping on the key transaction field based on the target field to generate various standard fields.

[0068] Optionally, the device also includes an alarm triggering module, used to: perform a full parsing test based on standard fields and calculate the parsing success rate; and trigger an alarm when the parsing success rate is lower than a preset threshold.

[0069] Optionally, the indicator and log storage module 330 specifically includes: a business indicator calculation unit, used to: extract, verify and calculate data for each standard field according to a preset period to generate various business indicators, including transaction volume, transaction time, transaction success rate, transaction response rate and error log rate.

[0070] Optionally, the metrics and log storage module 330 specifically includes: a metrics and log storage unit, used to: establish a metrics index and a tracking index; store business metrics in the metrics index and set a first specified period; store raw logs in the tracking index and set a second specified period, wherein the first specified period is longer than the second specified period.

[0071] The technical solution of this invention, by generating key transaction fields, enables unified parsing of logs in different formats, ensuring the success rate and consistency of extraction, and intelligently mining non-predefined fields to enrich data dimensions. Through fixed field mapping, the naming and format of fields across systems can be unified, establishing a globally standardized field system. By storing metrics and raw logs through a two-layer storage architecture, unified intelligent calculation of business metrics can be achieved, ensuring the accuracy and reliability of metrics. It also enables differentiated data storage management, balancing long-term retention and cost optimization, and improving data access efficiency.

[0072] The business log indicator data cleaning device provided in this embodiment of the invention can execute the business log indicator data cleaning method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0073] Example 4 Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0074] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0075] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a business log metric data cleaning method.

[0077] In some embodiments, a business log metrics data cleaning method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the business log metrics data cleaning method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a business log metrics data cleaning method by any other suitable means (e.g., by means of firmware).

[0078] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0080] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0083] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0084] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for cleaning business log indicator data, characterized in that, include: Obtain the original business logs, parse the original business logs, and generate key transaction fields; The key transaction fields are mapped to fixed fields to generate standard fields; Business metrics are calculated based on the standard fields, and the business metrics and the original logs are stored using a two-tier storage architecture.

2. The method according to claim 1, characterized in that, The process of obtaining the original business logs includes: Obtain a standardized template; The system to be cleaned is configured using a standardized template to obtain the original business logs.

3. The method according to claim 2, characterized in that, The step of parsing the original business logs to generate key transaction fields includes: Determine the format type of the original business logs and determine the preset priority for each format type; The original business logs are parsed according to a preset priority to extract key transaction fields, including a unique serial number, a global serial number, a transaction status, and a transaction duration.

4. The method according to claim 1, characterized in that, The step of mapping the key transaction fields to fixed fields to generate standard fields includes: Obtain a fixed field template, wherein the fixed field template includes various fixed fields; Determine the target field corresponding to the key transaction field from the fixed field template; Based on the target field, the key transaction fields are mapped to fixed fields to generate standard fields.

5. The method according to claim 4, characterized in that, The method further includes: A full parsing test was performed based on the aforementioned standard fields, and the parsing success rate was calculated. An alarm is triggered when the parsing success rate is lower than a preset threshold.

6. The method according to claim 1, characterized in that, The calculation of business metrics based on the standard fields includes: Data is extracted, verified, and calculated for each standard field according to a preset cycle to generate various business indicators, including transaction volume, transaction time, transaction success rate, transaction response rate, and error log rate.

7. The method according to claim 1, characterized in that, The storage of the business metrics and the raw logs based on the dual-layer storage architecture includes: Establish indicator indexes and tracking indexes; The business metrics are stored in the metric index and a first specified period is set. The original logs are stored in the tracking index and a second specified period is set, wherein the first specified period is longer than the second specified period.

8. A business log indicator data cleaning device, characterized in that, include: The key transaction field generation module is used to obtain the original business logs, parse the original business logs, and generate key transaction fields. The standard field generation module is used to perform fixed field mapping on the key transaction fields and generate standard fields. The metrics and log storage module is used to calculate business metrics based on the standard fields and store the business metrics and the original logs based on a two-layer storage architecture.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions that are used to cause a processor to execute the method of any one of claims 1-7.

Images