Business abnormity early warning method based on log semantic recognition and pattern matching

Abstract

The invention discloses a service exception early warning method based on log semantic recognition and pattern matching. According to the method, original logs are collected and preprocessed to obtain a standardized log stream, a suspected abnormal log set is screened out through an abnormal feature rule base containing keywords, regularities and frequency rules, a large language model is input to analyze semantics, and a structured result is output. And the result is stored in the exception knowledge base and indexed, and the new log and the database are subjected to semantic matching, so that the accuracy of the service exception early warning is improved, and the problem of low accuracy of the service exception early warning caused by low interpretability of the exception detection result in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent operation and maintenance, and particularly relates to a business exception early warning method based on log semantic recognition and pattern matching. BACKGROUND

[0002] Firstly, by means of distributed log collection technology, the original log data generated by the servers, databases, application programs and other nodes in the business system are real-time captured, and at the same time, the data cleaning technology is used to pre-process the logs, remove invalid characters, unify the field format, and convert the unstructured text logs into semi-structured data; then, the semantic recognition technology in the field of natural language processing is adopted, and the pre-processed logs are deeply analyzed by combining the pre-training language model, the key information such as business module, operation type and return code is extracted through entity recognition, the event correlation in the logs is combed by using relation extraction, and a structured log semantic feature library is constructed; subsequently, the pattern matching engine is started, the AC automaton multi-pattern matching algorithm and the fuzzy matching algorithm are integrated, the extracted log semantic features are compared with the preset abnormal pattern library (including the feature patterns extracted from the historical abnormal cases, and the risk threshold patterns set based on the business rules) in real time, and at the same time, the machine learning algorithm (such as decision tree and long short-term memory network) is introduced to analyze the trend of the log data and mine the potential unknown abnormal patterns.

[0003] When the semantic features and the abnormal patterns are matched, or the log data exceeds the normal fluctuation range, the system pushes the abnormal details (including log source, abnormal type, semantic analysis result and risk level) to the operation and maintenance management platform through the message pushing technology (such as SMS interface), synchronously calls the log visualization technology (such as Kibana and Grafana) to generate the abnormal timeline, the associated log chain and the influence range atlas, assists the operation and maintenance personnel to quickly locate the problem source, starts the emergency response process, and realizes the timely early warning and disposal of business exceptions.

[0004] For example, the invention patent with publication number CN106250303A discloses a business log collection and early warning system and method, which comprises: a receiving part for acquiring configuration information, the configuration information including the to-be-received business log types and the receiving rules, the to-be-received business log types being a subset of the preset business log type set, the business log being received according to the configuration information and stored in the search server according to the preset elements; a configuration part for acquiring the early warning rules; a scheduling part for calling the early warning task according to the early warning rules; an analysis part for analyzing the early warning task and statistically analyzing the business log stored in the search server according to the analyzed early warning task to generate the early warning result stored in the database; and a display part for displaying the early warning result. When the to-be-received business log types need to be changed, only the new configuration information needs to be input by the receiving part, and the business log collection and storage can be performed according to the new configuration information.

[0005] For example, the invention patent with publication number CN115168165A discloses a log-based business alarm method and device, which comprises: obtaining a preset number of sample log sets with unified format from the application to be detected; performing deduplication, filtering, log field conversion and information tag addition processing on the preset number of sample log sets with unified format to generate a processed sample log set; and performing early warning on the processed log data set according to the pre-set early warning condition. The present application obtains a preset number of sample log sets with unified format from the application to be detected; performs deduplication, filtering, log field conversion and information tag addition processing on the preset number of sample log sets with unified format to generate a processed sample log set; and performs early warning on the processed log data set according to the pre-set early warning condition.

[0006] The above-mentioned technology at least has the following technical problems: During the operation of the business system, a large amount of log data will be generated, and the log formats are not unified (covering unstructured text output by multiple nodes such as servers, databases and application programs). The existing technology lacks an efficient log preprocessing and screening mechanism. If the full log is directly analyzed, the data processing link will be long, it is impossible to quickly focus on suspected abnormal information, and thus the abnormal detection response will be delayed, which makes it difficult to meet the real-time warning needs of the business system; for potential abnormalities that do not directly appear in the log but can be judged through context association, the existing technology cannot accurately identify them, resulting in missing of abnormal detection; Attempts have been made to use semantic analysis models to deeply process logs, but no pre-screening of log data is performed, and full logs are directly input into the semantic analysis model. Due to the large amount of full log data, the calculation complexity of the semantic analysis model (especially deep learning models) is high, which will occupy a large amount of computing resources, resulting in low model running efficiency and causing serious response delay problems. It is difficult to balance performance overhead and intelligent analysis capability, and it is difficult to apply to business scenarios with strict requirements on resource occupation and response speed; when an abnormality similar to a historical case occurs, it is impossible to reuse existing experience through fast matching, and complex analysis steps still need to be repeated, resulting in low abnormality processing efficiency; at the same time, the existing technology does not have the ability to automatically optimize detection rules and models according to new abnormal types; the existing abnormality detection technology relying on keywords or simple rules can only identify the existence of abnormalities, but cannot accurately classify abnormal types (such as network timeout and database connection abnormality), and cannot extract abnormal root causes and provide targeted repair suggestions, which causes the problem of low accuracy of business abnormality warning due to low explainability of abnormality detection results. SUMMARY

[0007] The embodiment of the application provides a business exception early warning method based on log semantic recognition and pattern matching, and solves the problem of low business exception early warning accuracy caused by low exception detection result interpretability in the prior art, and improves the business exception early warning accuracy.

[0008] The embodiment of the application provides a business exception early warning method based on log semantic recognition and pattern matching, and solves the problem of low business exception early warning accuracy caused by low exception detection result interpretability in the prior art, and improves the business exception early warning accuracy. The one or more technical solutions provided in the embodiment of the application have at least the following technical effects or advantages: 1. By constructing a multi-dimensional abnormal feature rule library and optimizing the standardized preprocessing process, the core pain points of "massive data redundancy" and "abnormal feature omission" in traditional log processing are effectively solved. In the preprocessing stage, by unifying the time stamp, extracting the module identifier and cleaning the noise information, the unstructured original log is converted into a standardized log stream, providing a unified data basis for subsequent screening and avoiding recognition bias caused by format disorder. In the abnormal feature rule library, the combination of the basic keyword set and the business self-defined keyword set covers both general abnormal scenarios and personalized needs of different business modules, and the semantic verification mechanism further ensures the uniqueness and effectiveness of the keywords; the design of automatically generating regular expressions based on log samples reduces the technical threshold of rule configuration, and the accuracy threshold verification ensures the accuracy of rule matching; the sliding time window and dynamic threshold strategy of the frequency abnormality detection rule can accurately capture the sudden abnormality of log frequency and record detailed background information, providing complete context for subsequent analysis.

[0009] 2. By relying on the natural language understanding ability of the large language model and the refined analysis logic, this method breaks through the limitation of traditional abnormality recognition that can only judge "whether abnormal" and realizes "precise classification of abnormal types, deep positioning of root causes and executable processing suggestions" of the whole link analysis. The customized prompt template guides the model to analyze the log according to the unified standard through clear analysis instructions, classification reference table and output constraints, ensuring the consistency and traceability of abnormal type identification; the three-level attribution logic gradually digs from direct phenomena, associated factors to potential risks, and combines cross-module log analysis associated with tracking identifiers to completely restore the causes and propagation paths of the abnormality, avoiding the one-sidedness of traditional analysis "only seeing the trees but not the forest"; the structured output result contains key information such as abnormal type, root cause and processing suggestion, so that the operation and maintenance personnel can quickly master the core situation of the abnormality without manually checking massive logs.

[0010] 3. Through the structured storage of the abnormal knowledge base and the semantic similarity matching mechanism, the method realizes efficient reuse of historical abnormal experience and autonomous evolution of system capability. The knowledge base adopts the design of "classified storage + multi-dimensional indexing", which stores the abnormal type, cause description and semantic vector in association, and combines with the FAISS vector indexing technology to ensure that new abnormal logs can be quickly matched with historical data. The identification response time of known abnormalities is controlled within milliseconds, realizing the efficient early warning mode of "one-time analysis, multiple reuse". The feature extraction and standardization processing before matching of new logs ensure the compatibility with knowledge base data, avoiding matching deviation caused by expression differences; the two-stage matching and dynamic threshold adjustment strategy balances the matching speed, continuously optimizes the threshold parameters through accuracy feedback, so that the matching accuracy is always maintained at a high level. In addition, the analysis results of new abnormalities are automatically updated to the knowledge base, synchronously optimizing the screening rules and model prompt templates, forming a closed-loop self-learning mechanism of identification-storage-reuse-optimization. With the continuous operation of the system, the abnormal knowledge base is continuously enriched, the identification period of unknown abnormalities is gradually shortened, the coverage and accuracy of system early warning are continuously improved, and the system adapts to the changing abnormal scenarios of business systems, with strong expansibility and long-term applicability. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A business abnormality early warning method based on log semantic identification and pattern matching is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a business abnormality early warning method based on log semantic identification and pattern matching, which solves the problem of low business abnormality early warning accuracy caused by low explainability of abnormal detection results in the prior art. The method includes collecting original logs and preprocessing to obtain standardized log streams, screening a suspected abnormal log set using an abnormal feature rule library containing keywords, regular expressions and frequency rules, inputting a large language model to analyze semantics and outputting structured results. The results are stored in an abnormal knowledge base and indexed, and new logs are matched with the library based on semantics, thereby improving the accuracy of business abnormality early warning.

[0013] The technical solutions in the embodiments of the present application are used to solve the problem of low business abnormality early warning accuracy caused by low explainability of abnormal detection results, and the general idea is as follows: The original log data generated in the running process is collected in real time, the original log data is preprocessed, standardized log stream data is obtained, an abnormal feature rule library is constructed, the abnormal feature rule library includes keyword-based matching rules, regular expression-based pattern matching rules, and frequency anomaly detection rules, the standardized log stream data is input to step two, the suspected abnormal log set is input to the pre-set large language model, the log semantics are analyzed through the natural language understanding capability of the large language model, abnormal pattern recognition is realized, and structured results are output, the structured results output by step three are stored in the abnormal knowledge base in the form of abnormal types, reason descriptions, and semantic vectors and are indexed, and new abnormal logs received are matched with the execution semantics in the abnormal knowledge base, thereby improving the business abnormality early warning accuracy.

[0014] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0015] As shown in Figure 1 Fig. 1 is a flow chart of a business abnormality early warning method based on log semantic recognition and pattern matching provided by an embodiment of the present application. The method comprises the following steps: step one, collecting original log data generated in the running process in real time, and preprocessing the original log data to obtain standardized log stream data; step two, constructing an abnormal feature rule library, the abnormal feature rule library including keyword-based matching rules, regular expression-based pattern matching rules, and frequency anomaly detection rules, inputting the standardized log stream data to step two, and screening the standardized log stream data by the abnormal feature rule library to obtain a suspected abnormal log set, the suspected abnormal log set being used to represent log entries with abnormal features identified from massive logs; step three, inputting the suspected abnormal log set to a pre-set large language model, analyzing log semantics through the natural language understanding capability of the large language model, realizing abnormal pattern recognition, and outputting structured results; and step four, storing the structured results output by step three in the abnormal knowledge base in the form of abnormal types, reason descriptions, and semantic vectors and indexing, and matching new abnormal logs received with execution semantics in the abnormal knowledge base.

[0016] It should be noted that the application innovatively proposes a "rule screening, semantic recognition" double-layer architecture, which first realizes efficient abnormal candidate filtering by using keywords and statistical patterns, and then performs deep semantic understanding on the candidate logs by a large language model. The structure greatly reduces the semantic model calling amount while maintaining the real-time detection, and balances the performance and intelligence. The large language model is introduced to perform semantic level pattern recognition on the log content through natural language understanding capability. The model can automatically identify abnormal types (such as network timeout, database connection exception, permission verification failure, etc.) according to context information, and induce abnormal root causes and possible repair suggestions, realizing the explainability of the abnormal detection result. On the basis of the semantic analysis result, an abnormal knowledge base is constructed, and the abnormal types and cause patterns recognized by the model are stored in a patternized manner. Subsequently, known abnormalities can be quickly identified through semantic vector matching, realizing "experience reuse" and "automatic optimization". At the same time, the system can automatically update the screening rules and model prompt templates according to the newly added abnormal types, and has self-learning ability. The stream processing architecture (such as based on Kafka / Flink) is adopted to realize real-time collection and screening of logs, supports multi-module configuration of keyword strategy, frequency anomaly detection, context association analysis and other functions, and can efficiently run in an enterprise-level distributed environment, and has good scalability and universality.

[0017] It should be understood that the log data generated during the operation of the system is accessed in real time by the collection module, and the log is time-stamped, sorted, module-identified, and content-cleaned to form a standardized log stream input. The log information containing abnormal features is matched through a rule library. The rule library can include keyword matching (such as "error", "exception", "timeout", "fail", etc.), regular expression matching, and frequency mutation detection based on a statistical model. The module outputs a "suspected abnormal log set" and labels the trigger cause and log context. The filtered suspected abnormal logs are input into a large language model for semantic recognition and pattern matching. The model understands the log semantics based on a pre-set prompt template (Prompt) and outputs structured results. The model output includes: abnormal type (such as database exception, network exception, business logic error, etc.); root cause analysis (such as connection pool depletion, dependent service response timeout, parameter mismatch, etc.); processing suggestions (such as restarting the service, checking the configuration, optimizing the transaction logic, etc.), at the same time, the model can summarize related logs according to the same trace_id or request ID, analyze the causal relationship between multiple modules, and form a complete abnormal link description. The structured results output by the model are stored in the abnormal knowledge base, and are indexed in the form of abnormal type, cause description, semantic vector, etc. When new abnormal logs appear in the future, the system first performs semantic similarity matching in the knowledge base, if the matching is successful, the corresponding abnormal type and processing suggestion are directly output, if the matching fails, the log is sent to the large language model for reanalysis and the knowledge base is updated; according to the severity and impact range of the model output, the system sends warning information to the monitoring platform through the message queue or interface. The alarm information can include the abnormal type, root cause summary, involved services, and recommended handling measures. At the same time, the user's confirmation or feedback of the processing result will update the knowledge base in reverse, improving the accuracy of subsequent analysis.

[0018] In the embodiment, the business anomaly early warning method realizes the collaborative landing of multi-dimensional technical advantages through the "preprocessing-screening-analysis-storage matched full-process closed-loop design. In the log processing front end, the standardized preprocessing eliminates the data format differences, combined with the multi-rule screening mechanism containing keywords, regular expressions and frequency detection, it can efficiently and accurately lock the suspected abnormal items from massive logs, greatly improve the efficiency and accuracy of abnormal screening, and effectively solve the problems of redundant data interference and missing abnormal characteristics in traditional log processing; in the core analysis link, relying on the customized large language model, it deeply mines the log semantic information, realizes the accurate classification of abnormal types, the hierarchical tracing of root causes and the generation of executable processing suggestions, breaks the limitation of traditional abnormal identification that can only judge the existence of abnormality, significantly reduces the problem positioning and decision-making cost of operation and maintenance personnel; in the long-term optimization level, through the structured storage to build a dynamic abnormal knowledge base and establish multi-dimensional index, combined with the semantic similarity matching mechanism to realize the rapid reuse of historical abnormal experience, at the same time, through the continuous inclusion of new abnormal data to form a self-learning closed loop, the system early warning ability is continuously iterated and upgraded with the operation of the business, which not only guarantees the high-precision early warning demand in the current complex business scenario, but also has good expansibility and long-term applicability to adapt to the future business changes, and fully improves the automation and intelligence level of intelligent operation and maintenance.

[0019] Further, the specific steps of obtaining standardized log stream data are: by real-time access to multi-node log data generated in the operation process of the business system, adding a unified format timestamp to each log record, extracting and identifying the specific system or service module that generates the log, and cleaning the log content to remove irrelevant characters or noise to remove invalid information and unify the data format, to obtain standardized log stream data.

[0020] In the embodiment, the acquisition of the standardized log stream data is processed by multi-dimensional refinement, which realizes the normalization and value extraction of the original log data, and achieves significant technical effects: first, the design of real-time access to multi-node log data ensures the integrity of the business system full-link log capture, avoiding breakpoints in subsequent abnormal analysis due to log loss; second, a unified format timestamp is added to each log, solving the problem of inconsistent log time expression across different nodes, providing a precise time reference for subsequent tracing of abnormal timing and analyzing event causality; extracting and identifying specific system or service modules realizes the precise positioning of log sources, facilitating quick locking of abnormal business scope and reducing the blindness of cross-module troubleshooting; the log content cleaning step effectively reduces data redundancy by removing irrelevant characters, noise and invalid information, and unifies the data format to eliminate differences in log formats across different modules, providing a standardized and high-quality data foundation for efficient screening of the abnormal feature rule library and accurate semantic analysis of the large language model, improving the processing efficiency and analysis accuracy of the entire abnormal early warning process from the source, and avoiding problems such as screening bias and analysis error caused by disordered data.

[0021] Further, the specific steps of constructing the abnormal feature rule library are as follows: the keyword-based matching rule includes a basic abnormal keyword set and a business-defined keyword set, the basic abnormal keyword set predefines abnormal keyword core vocabulary; the business-defined keyword set supports users to add, delete or modify keywords through a visual configuration interface according to the characteristics of the business module, and performs semantic verification on the newly added keywords to avoid semantic repetition or conflict with existing keywords, and associates each keyword with a corresponding abnormal scenario label for subsequent scenario classification and labeling of suspected abnormal logs.

[0022] In the present embodiment, the construction step of the keyword matching rule in the abnormal feature rule library is realized through the cooperative design of the basic and custom keyword sets, achieving the comprehensiveness of rule coverage and the flexibility of business adaptation, and achieving significant technical effects: the basic abnormal keyword set presets core words, ensuring the all-around coverage of general abnormal scenarios, providing stable and reliable basic rule support for the preliminary screening of massive logs; supporting users to customize keywords based on the characteristics of business modules through a visual interface, greatly reducing the operation threshold of non-technical personnel, enabling the rules to accurately match the individualized abnormal features of different businesses, effectively solving the pain points of traditional fixed rules that are difficult to adapt to diversified business scenarios; the semantic verification mechanism of the new keyword avoids the confusion caused by semantic repetition or conflict, ensuring the standardization and rigor of the rule library; and the design of associating each keyword with an abnormal scenario label realizes the accurate classification and labeling of suspected abnormal logs, providing scenario-based guidance for subsequent semantic analysis of large language models, helping to quickly locate the domain of the abnormal, significantly improving the pertinence and efficiency of the overall abnormal identification, and laying a high-efficiency rule foundation for the subsequent depth analysis link.

[0023] Further, the pattern matching rule based on regular expressions is designed for abnormal logs with fixed format characteristics; the regular expression is automatically generated based on the log format sample, the fixed field and variable field in the abnormal log sample are extracted by inputting the same type of abnormal log sample, the corresponding regular expression rule is automatically generated, the user can input the test log to verify the rule matching accuracy; if the matching accuracy is lower than the preset accuracy threshold, the matching logic of the variable field is prompted to be adjusted; if the matching accuracy is higher than or equal to the preset accuracy threshold, no adjustment is prompted.

[0024] In the embodiment, based on the regular expression-based pattern matching rule construction mode, the efficiency of rule configuration and the accuracy of matching are realized through the cooperative design of automatic generation and accurate verification, and the following remarkable technical effects are achieved: the rules are specially designed for abnormal logs with fixed format characteristics, which ensures the accurate capture of this kind of standardized abnormality and solves the problem that the traditional general rules are not sensitive to the identification of abnormality with specific format; the fixed and variable fields are automatically extracted based on the same type of log samples and regular expressions are generated, which greatly reduces the workload of manually writing rules, reduces the dependence on the professional ability of technical personnel, and significantly improves the rule configuration efficiency; the matching accuracy of the rules can be tested by inputting test logs, and the deviation of the matching logic of the variable fields in the rules can be found and corrected in time through the preset threshold prompt adjustment function, so that the rule matching accuracy is always maintained at the expected level, and the abnormality misjudgment or misjudgment caused by rule design defects is effectively avoided; the overall design not only guarantees the specific identification ability of the rules for abnormality with specific format, but also takes into account the configuration efficiency and matching accuracy through the automation and verification mechanism, provides high-quality exclusive rule support for the abnormal feature rule library, and further improves the reliability and effectiveness of the mass log screening link.

[0025] Further, the frequency abnormality detection rule is implemented based on a sliding time window and a statistical threshold value; by setting the sliding time window length and the log statistical dimension, the frequency fluctuation range of normal logs under each statistical dimension is calculated through historical log data, and the mean value of the frequency fluctuation range is taken as the initial frequency abnormality threshold value; if the log frequency in the sliding time window is greater than the initial frequency abnormality threshold value, it is determined that there is frequency abnormality; if the log frequency in the sliding time window is less than or equal to the initial frequency abnormality threshold value, it is determined that there is no frequency abnormality; through the dynamic threshold value updating strategy, the threshold value is automatically updated based on historical data at a preset time point, and the frequency abnormality detection rule records the log details in the time window at each abnormality triggering time, including the number of log records, the peak occurrence time, the associated business module, and stores the log details in the time window and the log content into a suspected abnormal log set to provide frequency abnormality background information.

[0026] In the embodiment, the frequency anomaly detection rule is designed by sliding time window and dynamic threshold, which realizes accurate capture and all-scene adaptation of log frequency anomaly, and achieves significant technical effects: the flexible setting of sliding time window accurately divides the statistical dimension, which can focus on the log frequency change of specific time range and business dimension, avoiding the lag and limitation of traditional fixed time interval detection; the normal frequency fluctuation range is calculated based on historical log data, and the mean value is used as the initial threshold, which ensures the scientificity and rationality of threshold setting, and conforms to the actual operation law of business system; the threshold is automatically updated at the preset time point through dynamic adjustment strategy, so that the rule can adapt to the periodic change or sudden fluctuation of business traffic, effectively solving the problem that fixed threshold is difficult to adapt to the dynamic running state of the system; at the same time, the log details (number, peak time, associated module) of each abnormal trigger are recorded and included in the suspected abnormal log set, which provides rich background information for subsequent abnormal analysis and helps to quickly trace the root cause of abnormality; the overall design not only realizes efficient identification of frequency mutation anomaly, but also guarantees the accuracy and robustness of detection through dynamic threshold optimization, further improves the screening ability of abnormal feature rule library, and provides more valuable candidate log data for subsequent deep semantic analysis.

[0027] Further, the specific steps of analyzing log semantics through the natural language understanding ability of the large language model are as follows: the preset prompt template of the large language model includes abnormal analysis guide instructions, abnormal type classification reference table and output format constraints; the abnormal analysis guide instructions require the large language model to analyze the suspected abnormal log text line by line, preferentially extract business modules, error features, key parameters and tracking identification information, and then judge the abnormal nature in combination with the abnormal type classification reference table; the abnormal type classification reference table presets the classification of abnormal types, and the output format constraints require the large language model to mark the abnormal type identification result and supplement the judgment basis, so as to ensure that the abnormal type identification result is traceable.

[0028] In the embodiment, the specific steps of analyzing log semantics by a large language model, with the fine design of customized preset prompt template, realize the standardization, precision and traceability of abnormal identification, and achieve significant technical effects: the abnormal analysis guide instructions integrated in the preset prompt template clearly define the core tasks and priorities of the model in parsing logs line by line, ensuring that the model can efficiently extract key information such as business modules and error features, and avoid the blindness of the parsing process; the abnormal type classification reference table provides a unified judgment standard for the model, effectively standardizing the classification results of abnormal types, and solving the problem of chaotic and inconsistent classification of abnormalities in traditional semantic analysis; the output format constraint requires labeling the identification results and supplementing the judgment basis, so that each abnormal identification has a trace, greatly improving the credibility and traceability of the results, and facilitating subsequent manual verification and problem review. The overall design guides and constrains the entire process of model analysis through the whole process, fully utilizes the natural language understanding advantages of large language models, ensures the efficiency and accuracy of abnormal semantic analysis, and realizes the standardized output of analysis results, laying a high-quality foundation for subsequent abnormal root cause analysis, processing suggestion generation and knowledge base storage, and significantly improving the professionalism and reliability of the entire business abnormal warning process.

[0029] Further, the log semantic analysis adopts a three-level attribution logic, and the specific process is as follows: the first level directly attributes the phenomenon to extract the direct error performance from the suspected abnormal log set and label the direct root cause; the second level associated factor mining, if the log does not explicitly indicate the direct root cause, the large prediction model traces the associated pre-log of the same business request, analyzes the system operation before the abnormality occurs, and labels the indirect root cause; the third level potential risk supplement, combined with the failure rule, supplement the potential influencing factors, and supplement the potential risks; the root cause analysis results are output in the order of direct root cause, indirect root cause and potential risk, and each level needs to be associated with the corresponding log segment as support.

[0030] In the embodiment, the three-level attribution logic realizes the deep mining and comprehensive presentation of the root cause of the exception through hierarchical progressive analysis design, and achieves the following remarkable technical effects: the first level directly attributes the phenomenon to focus on the explicit error performance in the log, providing preliminary positioning for the exception analysis; the second level of associated factor mining uses the tracking identifier to link the pre-log of the same business request, breaks through the information limitation of a single log, and traces the indirect root cause through analyzing the system operation before the exception occurs, solving the one-sidedness of traditional analysis which only stays on the surface phenomenon; the third level of potential risk supplement combines the failure law to predict potential influencing factors, realizes the forward-looking analysis from "occurred exception" to "potential risk", and enhances the predictability of the exception disposal. At the same time, each level is associated with the corresponding log segment as support to ensure the objectivity and traceability of the attribution result. The overall design not only guarantees the depth and breadth of root cause analysis, but also makes the operation and maintenance personnel clearly understand the origin and development of the exception and the potential risk through structured output, provides a comprehensive basis for formulating accurate and effective processing scheme, significantly improves the pertinence and efficiency of the exception disposal, and reduces the risk of decision-making errors caused by unclear root cause judgment.

[0031] Further, the specific steps of storing in the form of exception type, reason description, semantic vector and establishing index to the exception knowledge base are as follows: based on the output structured result, an exclusive index table is established for each exception type, the index table field includes exception type identifier, typical log feature, associated business module, historical occurrence frequency, the root cause analysis result is stored in three levels of direct cause, indirect cause and potential risk, keyword index is established for each level of cause; the exception type name and reason description text are converted, the vector storage is associated with the vector generation time and model version information, and a semantic vector index library is constructed, the vector distance calculation method is set as cosine similarity, the index library supports dynamic incremental update to ensure that the newly added semantic vector can be included in the matching range in real time.

[0032] In the embodiment, the storage and index construction of the abnormal knowledge base realizes efficient management and rapid reuse of abnormal knowledge through multi-dimensional structured storage and refined index design, and achieves significant technical effects: an exclusive index table is established for each abnormal type, including identification, typical characteristics and other fields, the root cause analysis results are split according to a three-level structure and a keyword index is established, which ensures the classification and accurate retrieval of abnormal information, and facilitates quick positioning of abnormal cases of specific types or root causes; the abnormal type name and cause description are converted into semantic vectors and associated with the generation time and model version information, and a semantic vector index library supporting dynamic incremental update is constructed by combining the vector distance calculation method of cosine similarity, which not only solves the limitations of traditional text matching and realizes deeper semantic association matching, but also ensures that newly added abnormal knowledge is included in the retrieval range in real time, adapting to the dynamic changes of the abnormal scene of the business system. The overall design takes into account the structured management and high-dimensional semantic retrieval requirements of abnormal knowledge, greatly improving the efficiency and accuracy of subsequent new abnormal log and knowledge base matching, laying a solid data foundation for realizing "experience reuse" and system self-learning, and significantly enhancing the long-term usability and expansion capability of the abnormal early warning system.

[0033] Further, the received new abnormal log and the abnormal knowledge base are executed for semantic similarity matching, which further includes: extracting abnormal text segments, business module identifiers, abnormal occurrence times and tracking identifiers in the new abnormal log, and eliminating redundant information in the log; performing format uniform processing on the extracted abnormal text segments to ensure that the text expression is consistent with the historical data stored in the abnormal knowledge base; using the same model as the semantic vector generation in the abnormal knowledge base, converting the standardized abnormal text segments, and taking the average of the vectors of each segment as the final semantic vector of the new log during the vector generation process, while recording the text source segment ID during vector generation to facilitate the tracing of the matching results.

[0034] In the embodiment, the new abnormal log preprocessing process lays a high-precision data foundation for semantic similarity matching through multi-dimensional standardization processing, achieving significant technical effects: extracting key information such as abnormal text segments and business modules and eliminating redundant content, achieving log data simplification and focusing, and avoiding irrelevant noise interference matching accuracy; format uniform processing of format text expression eliminates expression differences with historical data in the knowledge base, solving the matching deviation problem caused by the same semantics but different expressions; the same model is used for semantic vector conversion and the average value strategy to ensure the dimension consistency and semantic comparability of the new log vector and the knowledge base vector, and the vector fusion of multiple segments can better represent the overall abnormal characteristics of the log; and the recording of the text source segment ID provides a basis for the traceability verification of the matching result, enhancing the explainability of the matching process. The overall design processes the whole process of new log from information extraction to vector generation through standardization, maximally reduces the interference of data heterogeneity on semantic similarity matching, makes the subsequent matching result with the abnormal knowledge base more accurate and reliable, and provides high-quality data support for quickly reusing historical abnormal processing experience.

[0035] Further, the specific steps of receiving the new abnormal log and performing semantic similarity matching with the abnormal knowledge base are as follows: extracting core keywords from the standardized text of the new abnormal log, matching the keywords index of the reason description in the abnormal knowledge base, and taking the historical abnormal data as a candidate matching set; if the number of the candidate matching set is 0, it is directly determined that no historical abnormality is matched, and the large language model reanalysis process is entered; the semantic vector of the new abnormal log is calculated with the semantic vector of the historical abnormality in the candidate matching set, an initial similarity threshold is set, if there is a historical abnormality vector and the similarity of the new log vector is greater than or equal to the initial similarity threshold, it is determined that the matching is successful, and the abnormal type, reason description and processing suggestion corresponding to the historical abnormality are output; if the similarity of all candidate vectors is less than the initial similarity threshold, the similarity of the new log vector and all historical vectors in the abnormal knowledge base under the same abnormal category is calculated, the data with the highest similarity is taken as the reference matching result, and the similarity value is labeled and output.

[0036] In the embodiment, the semantic similarity matching process realizes the efficiency and accuracy of the new abnormal log and knowledge base matching through the two-stage design of "keyword preliminary screening + semantic vector precise matching", and achieves significant technical effects: first, the candidate set is screened out through the rapid matching of core keywords and knowledge base index, which effectively reduces the range of subsequent precise matching, greatly improves the overall matching efficiency, and avoids the resource consumption and delay caused by full vector calculation; when the candidate set is empty, the large language model is triggered for reanalysis, ensuring the timely identification and processing of new type of abnormality and avoiding missing unknown abnormality; the vector precise matching based on cosine similarity combined with the initial threshold judgment can accurately locate the historical cases highly similar to the new abnormality, quickly reuse the mature abnormal type, root cause and processing suggestion, and significantly shorten the abnormal disposal period; for the cases that do not reach the threshold, the optimal reference is further found under the same abnormal category and the similarity is labeled, which not only provides valuable reference information for operation and maintenance personnel, but also retains the decision flexibility. The overall design takes into account the matching speed and accuracy, and at the same time realizes the rapid response of known abnormality and the reasonable disposal of unknown abnormality through layered judgment logic, fully plays the experience reuse value of abnormal knowledge base, and effectively improves the practical performance and decision support capability of the business abnormality early warning system.

[0037] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) having computer usable program code embodied thereon.

[0038] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks. Figure 1 The means for carrying out the functions specified in one or more flows and / or blocks.

[0039] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0040] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0041] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments. 1

[0042] It is apparent that a person skilled in the art can make a variety of modifications and variations to the application without departing from the scope of the same. Thus, the application is intended to encompass all such modifications and variations as fall within the scope of the appended claims and their equivalents.

Claims

1. A business anomaly early warning method based on log semantic recognition and pattern matching, characterized in that, Includes the following steps: Step 1: Collect raw log data generated during operation in real time, and preprocess the raw log data to obtain standardized log stream data; Step 2: Construct an anomaly feature rule base. The anomaly feature rule base includes keyword-based matching rules, regular expression-based pattern matching rules, and frequency anomaly detection rules. Input standardized log stream data into Step 2. The anomaly feature rule base filters the standardized log stream data to obtain a set of suspected anomaly logs. The set of suspected anomaly logs is used to represent log entries with anomaly features identified from massive logs. Step 3: Input the suspected abnormal log set into the pre-set large language model, and use the natural language understanding ability of the large language model to parse the log semantics, realize abnormal pattern recognition, and output structured results; Step four: Store the structured results output in step three in the form of anomaly type, cause description, and semantic vector in the anomaly knowledge base and create an index. Then, perform semantic similarity matching between the received new anomaly logs and the anomaly knowledge base.

2. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The specific steps for obtaining standardized log stream data are as follows: By accessing multi-node log data generated during the operation of business systems in real time, a unified format timestamp is added to each log record, the specific system or service module that generated the log is extracted and identified, and the log content is cleaned to remove irrelevant characters or noise to remove invalid information and to unify the data format, resulting in standardized log stream data.

3. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The specific steps for constructing the anomaly feature rule base are as follows: Keyword-based matching rules include a basic abnormal keyword set and a business-defined keyword set, wherein the basic abnormal keyword set is preset with core abnormal keyword terms. The custom keyword set allows users to add, delete, or modify keywords through a visual configuration interface based on the characteristics of the business module. Newly added keywords are semantically validated to avoid semantic duplication or conflict with existing keywords. At the same time, each keyword is associated with a corresponding abnormal scenario tag for subsequent scenario classification and labeling of suspected abnormal logs.

4. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The pattern matching rules based on regular expressions are designed for exception logs with fixed format characteristics. Regular expressions are automatically generated based on log format samples. By inputting abnormal log samples of the same type, fixed and variable fields in the abnormal log samples are extracted, and corresponding regular expression rules are automatically generated. Users can input test logs to verify the accuracy of the rule matching. If the matching accuracy is lower than the preset accuracy threshold, a prompt will be made to adjust the matching logic of the variable field; If the matching accuracy is higher than or equal to the preset accuracy threshold, no adjustment prompt will be given.

5. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The frequency anomaly detection rule is implemented based on a sliding time window and a statistical threshold. By setting the duration of the sliding time window and the log statistics dimensions, the frequency fluctuation range of normal logs under each statistical dimension is calculated using historical log data, and the average of the frequency fluctuation range is used as the initial frequency anomaly threshold. If the log frequency within the sliding time window is greater than the initial frequency anomaly threshold, it is determined to be a frequency anomaly. If the log frequency within the sliding time window is less than or equal to the initial frequency anomaly threshold, it is determined that the frequency is not abnormal. By dynamically adjusting the threshold update strategy, the threshold is automatically updated based on historical data at preset time points. The frequency anomaly detection rules record the log details within the time window when each anomaly is triggered, including the number of log entries, the peak occurrence time, and the associated business modules. The log details and log content within the time window are stored in the suspected anomaly log set to provide background information on frequency anomalies.

6. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The specific steps for parsing log semantics using the natural language understanding capabilities of a large language model are as follows: The preset prompt template of the large language model includes anomaly analysis guidance instructions, anomaly type classification reference table, and output format constraints; The anomaly analysis guidance instruction requires the Big Prophet model to parse the suspected anomaly log text line by line, prioritize extracting business modules, error features, key parameters and tracking identification information, and then combine the anomaly type classification reference table to determine the nature of the anomaly. The anomaly type classification is based on a preset anomaly type reference table. The output format constraints require the large language model to label the anomaly type identification results and supplement the judgment criteria to ensure that the anomaly type identification results are traceable.

7. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The parsing of log semantics employs a three-level attribution logic, with the specific process as follows: The first level of direct phenomenon attribution extracts direct error manifestations from the suspected abnormal log set and marks the direct root cause; the second level of correlation factor mining, if the log does not clearly identify the direct root cause, the big oracle model analyzes the system operations before the anomaly occurred by tracking and identifying the preceding logs of the same business request, and marks the indirect root cause. The third level of potential risk supplementation, combined with the failure pattern, supplements potential influencing factors and potential risks; The root cause analysis results are output in the order of direct root cause, indirect root cause, and potential risk, and each level needs to be associated with a corresponding log segment as support.

8. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The specific steps for storing the anomaly knowledge base in the form of anomaly type, cause description, and semantic vector, and then establishing an index, are as follows: Based on the structured output results, a dedicated index table is created for each anomaly type. The index table fields include anomaly type identifier, typical log characteristics, associated business modules, and historical occurrence frequency. The root cause analysis results are split and stored in a three-level structure of direct cause, indirect cause, and potential risk, and a keyword index is created for each level of cause. The exception type name and cause description text are converted, and the vector is stored with the vector generation time and model version information. A semantic vector index library is built, and the vector distance calculation method is set to cosine similarity. The index library supports dynamic incremental updates to ensure that newly added semantic vectors can be included in the matching range in real time.

9. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The step of receiving new exception logs and performing semantic similarity matching with the exception knowledge base also includes: By extracting abnormal text fragments, business module identifiers, abnormal occurrence times, and tracking identifiers from new abnormal logs, redundant information in the logs is eliminated. The extracted abnormal text fragments are formatted in a standardized manner to ensure that the text description is consistent with the historical data stored in the abnormal knowledge base. Using the same model as the semantic vector generation in the anomaly knowledge base, the standardized anomaly text fragments are transformed. If the new log contains multiple anomaly text fragments during the vector generation process, the average value of each fragment vector is taken as the final semantic vector of the new log. At the same time, the text source fragment ID at the time of vector generation is recorded to facilitate the tracing of the matching results.

10. The business anomaly early warning method based on log semantic recognition and pattern matching as described in claim 1, characterized in that, The specific steps for receiving new anomaly logs and performing semantic similarity matching with the anomaly knowledge base are as follows: Extract core keywords from the standardized text of new anomaly logs and match them with the keyword index of cause descriptions in the anomaly knowledge base, using historical anomaly data as a candidate matching set; If the number of candidate matching sets is 0, it is directly determined as a historical anomaly that no match was found, and the process of re-analyzing the large language model is initiated. The semantic vector of the new anomaly log is compared with the semantic vector of historical anomalies in the candidate matching set using cosine similarity calculation. An initial similarity threshold is set. If the similarity between a historical anomaly vector and the new log vector is greater than or equal to the initial similarity threshold, the match is considered successful, and the anomaly type, cause description, and handling suggestions corresponding to the historical anomaly are output. If the similarity of all candidate vectors is less than the initial similarity threshold, the similarity between the new log vector and all historical vectors under the same anomaly category in the anomaly knowledge base is calculated. The data with the highest similarity is taken as the reference matching result, and the similarity value is labeled and output.

Images