Log alarm processing method and system and electronic equipment

Abstract

The embodiment of the invention provides a log alarm processing method and system and electronic equipment. The method is applied to the technical field of the Internet, and comprises the steps of collecting log data, performing data preprocessing, machine learning anomaly detection and knowledge retrieval, calling a three-module architecture of a large model generation processing strategy, and fusing a core technology of artificial intelligence, in this way, through multi-technology fusion processing, not only is full-process automation realized, but also the processing efficiency is improved. According to the invention, automatic processing of log alarm is realized, the fault processing precision and efficiency are improved, and the human input is reduced, so that the operation and maintenance cost and economic loss of an enterprise are remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a log alarm processing method and system and electronic equipment. BACKGROUND

[0002] In the field of Internet, the online problem processing of large projects is very complex. When the online system has a problem, it will cause the user to be unable to use and other serious consequences, causing significant economic losses. Therefore, when the online system has a fault, the system fault needs to be repaired in time.

[0003] The traditional fault processing means mainly have the following points: 1, building an automatic monitoring and alarm system; 2, building an automatic expansion and contraction strategy; 3, building an automatic disaster recovery and recovery strategy; 4, using automatic deployment and rollback technology, etc. The traditional fault processing means has realized the automation of problem processing in some scenes, but the recovery of the problem still needs to rely on technical personnel in most cases, so the fault processing efficiency is low. SUMMARY

[0004] The present application provides a log alarm processing method and system and electronic equipment, which solves the technical problem of low fault processing efficiency.

[0005] According to a first aspect of the present application, a log alarm processing method is provided. The method comprises: collecting log data and performing data preprocessing to obtain a to-be-detected log; performing feature detection on the to-be-detected log to obtain an abnormal vector, and performing retrieval in a pre-constructed or real-time constructed vector database according to the abnormal vector to obtain a retrieval result; accessing a general large model, generating a fault processing flow according to the retrieval result, and pushing to an approval platform.

[0006] As described above, the aspect and any possible implementation manner further provides an implementation manner, wherein the to-be-detected log is detected by a predetermined algorithm to extract abnormal features, and the abnormal features include abnormal service name, abnormal time and abnormal status code; and an abnormal vector is generated according to the abnormal features and corresponding log text.

[0007] In some possible implementation manners, the searching according to the abnormal vector in the vector database constructed in advance or in real time to obtain a searching result comprises: searching according to the abnormal vector in a vector knowledge base constructed in advance or in real time to obtain fault processing information, the fault processing information comprising historical alarms, processing suggestions and artificial feedbacks; wherein, the construction of the vector knowledge base comprises: constructing the vector knowledge base by historical log alarm records and unstructured knowledge data, the vector knowledge base containing historical alarms, processing suggestions and artificial feedbacks; and vectorizing data in the vector knowledge base to obtain a vector database.

[0008] In some possible implementation manners, the vector knowledge base updating process comprises: storing the fault processing flow in a temporary cache area and storing according to abnormal types; performing semantic vectorization processing on the fault processing flow in the temporary cache area by a timing task to obtain a vectorization processing result; removing the vectorization processing result by a cosine similarity algorithm to obtain a de-duplicated vectorization result; and writing the de-duplicated vectorization result into the vector knowledge base to update the vector knowledge base.

[0009] In some possible implementation manners, the searching according to the abnormal vector in the vector database constructed in advance or in real time to obtain a searching result comprises: calculating the cosine similarity of the abnormal vector and a vector in the vector knowledge base, and matching the most similar vector according to the cosine similarity result as the searching result; or, calculating the Euclidean distance of the abnormal vector and a vector in the vector knowledge base, and matching the most similar historical knowledge vector according to the Euclidean distance result as the searching result.

[0010] In some possible implementation manners, the feature detection on the to-be-detected log to obtain the abnormal vector further comprises: identifying an abnormal level of the abnormal type, calculating an abnormal threshold of the to-be-detected log by a sliding window algorithm according to a time sequence fluctuation range of the to-be-detected log; and when the abnormality greater than the abnormal threshold is detected for a continuous preset number of times, grading adjusting the sensitivity of the abnormal threshold in combination with the abnormal level, so as to obtain the abnormal vector.

[0011] According to a second aspect of the present disclosure, a log alarm processing system is provided. The system comprises: a data source module configured to collect log data and perform data preprocessing to obtain to-be-detected logs; a data management module configured to perform feature detection on the to-be-detected logs to obtain abnormal vectors, and search according to the abnormal vectors in a vector database constructed in advance or in real time to obtain a searching result; and a large language model module configured to access a general large model, generate a fault processing flow according to the searching result, and push the fault processing flow to an approval platform.

[0012] In some possible implementation manners, the data source module comprises a log collection unit and a data preprocessing unit, wherein the log collection unit is configured to collect log information through a predetermined log framework to obtain to-be-processed logs; and the data preprocessing unit is configured to perform format unification, structured processing and data cleaning on the to-be-processed logs to obtain to-be-detected logs.

[0013] In some possible implementation manners, the data management module comprises a machine learning unit and a retrieval enhancement generation unit, wherein the machine learning unit is configured to perform feature detection on the to-be-detected logs to obtain an abnormality vector; and the retrieval enhancement generation unit is configured to perform retrieval in a pre-constructed or real-time constructed vector database according to the abnormality vector to obtain a retrieval result.

[0014] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to the above when executing the program.

[0015] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method according to the first aspect and / or the second aspect of the present disclosure.

[0016] The present application has the following beneficial effects: through the three-module architecture of collecting log data, performing data preprocessing, machine learning anomaly detection, knowledge retrieval and calling a large model to generate a processing strategy, and by fusing the core technologies of artificial intelligence and processing through multiple technologies, not only full-process automation is realized, log alarm automatic processing is realized, fault processing accuracy and efficiency are improved, and human input is reduced, thereby significantly reducing enterprise operation and maintenance costs and economic losses.

[0017] It should be understood that the content described in the summary section is not intended to limit the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail some embodiments with reference to the attached drawings. The accompanying drawings are used to better understand the present disclosure and do not limit the present disclosure. In the drawings, the same or similar reference numerals refer to the same or similar elements, and:

[0019] Figure 1 An example diagram of a log alarm processing system architecture capable of implementing embodiments of the present disclosure is shown;

[0020] Figure 2A flow chart of a log alarm processing method according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A block diagram of a log alarm processing system according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] To make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present disclosure.

[0024] In addition, the term "and / or" herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0025] In the field of Internet, the online problem processing of large projects is very complex. When the online system has a problem, it will cause the user to be unable to use and other serious consequences, causing significant economic losses. Therefore, when the online system has a fault, the system fault needs to be repaired in time.

[0026] Therefore, the present application adopts a three-module architecture of collecting log data and performing data preprocessing, machine learning anomaly detection and knowledge retrieval, and calling a large model to generate a processing strategy, and fuses the core technologies of artificial intelligence to process through multiple technology fusion. Not only does it realize full-process automation, realize log alarm automatic processing, improve fault processing accuracy and efficiency, but also reduces manpower investment, thereby significantly reducing enterprise operation and maintenance costs and economic losses.

[0027] Next, the system architecture of the present application will be described in detail. Please refer to Figure 1 , Figure 1 The log alarm processing system architecture example shown in the embodiments of the present application is shown in FIG. 1. The log alarm processing system architecture includes a data source layer, a data management layer, and a large language model layer (LLM Agent, Large Language Model).

[0028] The data source layer includes a data collection unit (a log collection unit) and a big data platform (a data preprocessing unit). In the development process of the production system of an enterprise, in order to quickly locate an online fault, a log system is constructed, and common frameworks include zap (a high-performance structured log framework), logrus (a structured log framework), etc. After source log data is collected from the log system, the source log data is aggregated to the big data platform by a tool such as Filebeat (a lightweight log collector) for unified storage of data. In the big data platform, the collected source log data is preliminarily processed, including unified data format, structured data processing, data cleaning, etc.

[0029] The data management layer is a secondary processing of data, including a machine learning processing module and a retrieval augmented generation (RAG) module. The machine learning processing module extracts features and detects abnormalities from log data by using algorithms such as time series and Bayesian. Key information in the log is extracted, including abnormal keywords, time, service, error information, etc. A trained model is used to determine whether the log content is abnormal, and abnormal information, abnormal level, key context information, etc. are output. The abnormal information is processed into a query vector as the input of the RAG module. Event structured semantic understanding (such as abnormal type, abnormal service, etc.) is used as the input of the LLM Agent module. A local knowledge base is constructed by the RAG module, and the data source includes historical alarms, processing suggestions, artificial feedback information, etc. After all the knowledge content is vectorized, it is stored in a vector database. The abnormal query vector of the ML component is accepted in the vector database for vector retrieval, and the related knowledge vector (historical processing method) of the abnormality is returned. The retrieved knowledge is structured into a prompt and input into the LLM Agent for reasoning and decision-making.

[0030] The large language model layer (LLM Agent layer) is used to access general large models such as deepseek, qwen and chatglm. The context is constructed by receiving abnormal information input from the ML and strategy input from the RAG module, and the general large model module is called to generate problem analysis information, processing logic or processing scripts, and push them to the approval platform.

[0031] In summary, although the traditional fault handling means is partially automated, the recovery relies on technical personnel, and the experience of common faults cannot be effectively precipitated, and the online problem handling of large projects is complex, which is easy to cause economic losses. The existing scheme only focuses on a single technology (such as only using machine learning or only using a rule engine), lacks the synergistic effect of cross-technology stacks, and is difficult to achieve such comprehensive automated processing and knowledge management.

[0032] The present application can integrate machine learning, retrieval enhancement generation and large language model technology, perform anomaly detection and feature extraction through a machine learning component, retrieve historical knowledge through a retrieval enhancement generation technology, and generate a processing strategy based on context through a large model technology to form an automated decision-making process. Specifically, by coordinating machine learning algorithms with large language model technology, the accuracy of anomaly detection is improved. Machine learning captures abnormal features in logs through algorithms such as Time Series, and the RAG module retrieves similar historical cases based on semantic vectors, reducing the positioning time of abnormal data and improving the efficiency of anomaly detection, thereby realizing automated processing of log alerts, improving fault handling accuracy and efficiency, and also reducing manpower, thereby significantly reducing enterprise operation and maintenance costs and economic losses.

[0033] Please refer to Figure 2 , Figure 2 A flowchart 200 of a log alert processing method according to an embodiment of the present application is shown. As Figure 2 shown, the method includes at least steps S210 to S230, which are described in detail as follows:

[0034] Step S210, collecting log data and performing data preprocessing to obtain a log to be detected.

[0035] Among them, the log information of the application program is recorded, managed and output through zap (high-performance structured log framework) or logrus (structured log framework).

[0036] After collecting the source log data from the log system, the source log data is summarized to a big data platform through tools such as Filebeat (lightweight log collector) for unified storage of data. Filebeat is a lightweight log collection tool in the ecology, mainly used for efficient collection, processing and transmission of log data from servers, applications or devices. It belongs to the category of log collectors and aims to solve the problem of high resource consumption and complex configuration of traditional log collection tools.

[0037] Step S220, performing feature detection on the log to be detected to obtain an abnormal vector, and performing retrieval in a pre-constructed or real-time constructed vector database according to the abnormal vector to obtain a retrieval result.

[0038] In an embodiment of the present application, feature detection is performed on the log to be detected to obtain an anomaly vector, including: detecting the log to be detected by a predetermined algorithm to extract abnormal features, the abnormal features including abnormal service names, abnormal times, and abnormal status codes; and generating an anomaly vector according to the abnormal features and corresponding log texts.

[0039] As an example, the predetermined algorithm can be a Time Series algorithm and a Bayesian algorithm. The Time Series algorithm is a method for processing data arranged in chronological order, and the core goal is to capture the pattern (such as trend, seasonality, periodicity) of data changing over time and predict future values or detect anomalies based on historical patterns.

[0040] In the application of log anomaly detection, when the log data (such as system indicators, request frequency) is time series data, the Time Series algorithm can be used for:

[0041] Anomaly detection: by learning the time series features of normal patterns, identify abnormal points deviating from expectations (such as sudden surge in CPU usage); trend prediction: predict log indicators in the future (such as request volume), and discover potential risks (such as insufficient capacity) in advance; periodicity analysis: identify periodic patterns in logs (such as daily 2:00 scheduled task logs), and filter normal fluctuations.

[0042] As a specific example, the request delay of a certain microservice increases slightly every day from 10:00 to 11:00 in the past 7 days (business peak period), and the time series algorithm learns the "seasonal" pattern and marks it as normal fluctuation; if the delay suddenly increases 5 times at 10:00 one day, it is determined to be abnormal, and the abnormal feature is extracted.

[0043] The Bayesian algorithm updates the posterior probability based on the prior probability and new evidence based on Bayes' theorem, and the core idea is "continuously adjust the belief about an event based on existing knowledge and new data".

[0044] As a specific example, in log anomaly detection, log anomaly detection often faces "uncertainty" (such as multiple features in the log being abnormal at the same time, but the root cause cannot be determined), and the Bayesian algorithm can be used for: multi-feature fusion: consider multiple log features (such as error codes, response times, call chain depths), calculate joint probabilities, and avoid single-feature misjudgment; root cause analysis: model the causal relationship between features through Bayesian networks to infer the most likely abnormal root cause; uncertainty quantification: evaluate the reliability of the anomaly detection result (such as the confidence of the current anomaly is 95%) to assist manual decision-making.

[0045] As a specific example, when both "database connection timeout" and "cache hit rate drop" appear in the system log, the Bayesian network can learn the causal chain of "database failure → cache access increase → hit rate drop" through historical data, and thus infer that "database failure" is the more likely root cause, thereby extracting the abnormal features.

[0046] In practical applications, the two algorithms can be used in coordination, for example, in the process of log anomaly detection, the Time Series algorithm is used to monitor the API (Application Programming Interface) request volume every minute, and the ARIMA model is used to predict the normal range; when the request volume suddenly exceeds 2 times the upper limit of the prediction, an "abnormal traffic" alarm is triggered, thereby quickly discovering abnormal features. Through the Bayesian algorithm, the conditional probability of each feature is calculated to infer the root cause (such as "85% probability is caused by database slow query"). Thus, the root cause is accurately located, and the manual troubleshooting time is reduced.

[0047] However, in the process of troubleshooting, fluctuations during business peak hours are easy to be misjudged, so a dynamic threshold strategy can be used to reduce the false alarm rate through historical data statistics and adaptive adjustment.

[0048] In an embodiment of the present application, feature detection is performed on the log to be detected to obtain an abnormal vector, which further includes: identifying an abnormal level of the abnormal type, calculating an abnormal threshold of the log to be detected through a sliding window algorithm according to a time series fluctuation range of the log to be detected; when the abnormality greater than the abnormal threshold is detected for a continuous preset number of times, the sensitivity of the abnormal threshold is adjusted in stages combined with the abnormal level, thereby obtaining the abnormal vector. Through the dynamic threshold strategy, the time series analysis and the staged sensitivity adjustment are combined to make the abnormal detection more suitable for the actual business scenario and improve the system reliability.

[0049] In an embodiment of the present application, the abnormal vector is searched in a pre-constructed or real-time constructed vector database to obtain a search result, which includes: searching the abnormal vector in a pre-constructed or real-time constructed vector knowledge base to obtain fault handling information, the fault handling information including historical alarms, handling suggestions and manual feedback; wherein the construction of the vector knowledge base includes: constructing the vector knowledge base through historical log alarm records and unstructured knowledge data, the vector knowledge base containing historical alarms, handling suggestions and manual feedback; and vectorizing the data in the vector knowledge base to obtain a vector database. The RAG module stores the data such as historical alarms and handling suggestions in a vectorized manner to construct a dynamically updated local knowledge base. When the same type of abnormality (such as "database connection timeout") occurs again, the historical handling scheme can be recalled directly through vector search to avoid repeated mistakes.

[0050] As an example, the specific way of building a vector database includes: connecting log systems, ticket systems through APIs, and pulling historical data (such as alarm records in the past year) at regular intervals; using ETL tools to configure data synchronization tasks and set incremental collection strategies (only get new or updated data). For unstructured historical documents (such as offline fault handling manuals), import through templated tables. Remove duplicate data, filter outliers, and desensitize sensitive information, and parse the data into structured fields of "fault type-fault description-treatment step-one optimization suggestion"; splice the core fields of structured knowledge, generate dense vectors through large language model reasoning, and store them according to fault type as index to get vector database. Vectorize and store historical alarm and treatment suggestion data to build a dynamically updated local knowledge base. When the same type of exception (such as database connection timeout) occurs again, the historical treatment plan can be recalled directly through vector retrieval to avoid repeating the same mistakes.

[0051] As an example, when the extracted exception log represents a database connection timeout, the ML module processes the exception information into an exception vector, which is input to the RAG module. According to the exception vector, the vector knowledge base is searched in real time or pre-built to obtain fault handling information. Specifically, the fault handling information is: database connection timeout-firewall blocked port-check firewall rules-optimization suggestion. And according to the fault handling information, the fault is repaired. Through the cooperation of machine learning algorithms and large language model technology, the accuracy of anomaly detection is improved, thereby realizing log alarm automatic processing.

[0052] To avoid knowledge base redundancy caused by directly writing new knowledge (such as repeatedly storing similar treatment plans), the two-stage update ensures the simplicity and retrieval efficiency of the knowledge base through caching classification and deduplication.

[0053] Therefore, in an embodiment of the present application, the vector knowledge base update process includes: storing the fault handling process in a temporary cache area and storing it by exception type; performing semantic vectorization processing on the fault handling process in the temporary cache area through a timing task to obtain a vectorization processing result; removing duplicates from the vectorization processing result through a cosine similarity algorithm to obtain a deduplicated vectorization result; and writing the deduplicated vectorization result to the vector knowledge base to update the vector knowledge base. The newly generated treatment strategy is automatically stored in the database after being approved, so that the knowledge base continues to evolve with the development of the business.

[0054] As a specific example, the update process includes: obtaining an alarm record 2025-05-20 order service API timeout, error code 504 from the work order system. Extract the service name "order service", the fault description "API timeout", and parse the processing steps as "restart gateway service, check load balancing configuration"; generate a vector [0.23, -0.15,..., 0.47] using Sentence-BERT; store the vector with metadata (labels, time) in Milvus, with the index field being "fault type"; check that there is no similar knowledge, and directly store in the database, and in the future, if there are similar alarms, the knowledge can be recalled through vector retrieval. The newly generated processing strategy is automatically stored after being approved, so that the knowledge base continues to evolve with the development of the business.

[0055] In an embodiment of the present application, according to the abnormal vector, a search is performed in a pre-constructed or real-time constructed vector knowledge base to obtain a search result, including: calculating the cosine similarity of the abnormal vector and the vectors in the vector knowledge base, and matching the most similar vector according to the cosine similarity result as the search result; or, calculating the Euclidean distance of the abnormal vector and the vectors in the vector knowledge base, and matching the most similar historical knowledge vector according to the Euclidean distance result as the search result.

[0056] Step S230, access a general large model, generate a fault processing flow according to the search result, and push to an approval platform.

[0057] In an embodiment of the present application, the language model layer (LLM Agent layer) is used to access general large models including deepseek, qwen, chatglm, etc. By receiving abnormal information input from ML and strategy input from the RAG module to construct a context, the general large model module is called to generate problem analysis information, processing logic or fault processing flow, and pushed to the approval platform. After accessing the ChatGLM large model, a complete processing flow including root cause analysis, verification steps and script code can be generated according to the search result, which greatly improves the executability of the strategy compared with the traditional rule engine.

[0058] In Figure 2 In the technical solution shown, through log data collection and data preprocessing, a three-module architecture of machine learning anomaly detection and knowledge retrieval, and calling a large model to generate a processing strategy is used, and the core technologies of artificial intelligence are fused, through multi-technology fusion processing, not only the full-process automation is realized, the log alarm automatic processing is realized, the fault processing accuracy and efficiency are improved, and the human labor input is reduced, thereby significantly reducing the enterprise operation and maintenance cost and economic loss.

[0059] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0060] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0061] Figure 3 A block diagram 300 of a log alarm processing system according to an embodiment of the present disclosure is shown. Reference Figure 3 As shown, the system includes: a data source module 310, a data management module 320, and a large language model module 330. The data source module collects log data and performs data preprocessing to obtain the logs to be detected. The data management module performs feature detection on the logs to be detected, obtains anomaly vectors, and retrieves the results from a pre-built or real-time constructed vector database. The large language model module connects to a general-purpose large model, generates a fault handling process based on the retrieval results, and pushes it to the approval platform. This three-module architecture—collecting log data and performing data preprocessing, machine learning anomaly detection and knowledge retrieval, and calling a large model to generate processing strategies—integrates core artificial intelligence technologies. Through multi-technology fusion processing, it not only achieves full-process automation and automated log alarm processing, improving the accuracy and efficiency of fault handling, but also reduces manpower input, thereby significantly reducing enterprise operation and maintenance costs and economic losses.

[0062] In one embodiment of this application, the data source module 310 includes a log acquisition unit 3101 and a data preprocessing unit 3102. The log acquisition unit is used to collect log information through a predetermined log framework to obtain logs to be processed. The data preprocessing unit is used to perform format unification, structured processing and data cleaning on the logs to be processed to obtain logs to be tested.

[0063] In one embodiment of this application, the data management module 320 includes a machine learning unit 3201 and a retrieval enhancement generation unit 3202. The machine learning unit is used to perform feature detection on the log to be detected to obtain anomaly vectors. The retrieval enhancement generation unit is used to perform retrieval in a vector database built in advance or in real time based on the anomaly vectors to obtain retrieval results.

[0064] In an embodiment of the present application, the machine learning unit 3201 is configured to detect the to-be-detected log by a predetermined algorithm, extract abnormal features, and the abnormal features include abnormal service name, abnormal time, and abnormal status code; and generate an abnormal vector according to the abnormal features and corresponding log text.

[0065] In an embodiment of the present application, the machine learning unit 3201 is further configured to identify an abnormal level of an abnormal type, calculate an abnormal threshold of the to-be-detected log by a sliding window algorithm according to a time sequence fluctuation range of the to-be-detected log; and when an abnormality greater than the abnormal threshold is detected for a continuous preset number of times, adjust the sensitivity of the abnormal threshold in combination with the abnormal level, thereby obtaining the abnormal vector.

[0066] In an embodiment of the present application, the retrieval enhancement generation unit 3202 is configured to retrieve in a pre-constructed or real-time constructed vector knowledge base according to the abnormal vector, obtain fault handling information, and the fault handling information includes historical alarm, processing suggestion, and artificial feedback; wherein, the construction of the vector knowledge base includes: constructing the vector knowledge base by historical log alarm records and unstructured knowledge data, and the vector knowledge base contains historical alarm, processing suggestion, and artificial feedback; and vectorizing the data in the vector knowledge base to obtain a vector database.

[0067] In an embodiment of the present application, the retrieval enhancement generation unit 3202 is further configured to store the fault handling process into a temporary cache area and store by abnormal type; perform semantic vectorization processing on the fault handling process in the temporary cache area by a timing task to obtain a vectorization processing result; remove the vectorization processing result by a cosine similarity algorithm to obtain a de-duplicated vectorization result; and write the de-duplicated vectorization result into the vector knowledge base to update the vector knowledge base.

[0068] In an embodiment of the present application, the retrieval enhancement generation unit 3202 is further configured to calculate the cosine similarity of the abnormal vector and the vector in the vector knowledge base, match the most similar vector according to the cosine similarity result as a retrieval result; or, calculate the Euclidean distance of the abnormal vector and the vector in the vector knowledge base, match the most similar historical knowledge vector according to the Euclidean distance result as a retrieval result.

[0069] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0070] In the technical solution of the present application, the acquisition, storage, and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0071] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0072] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0073] The electronic device 400 includes a computing unit 401 that can perform various appropriate actions and processes according to a computer program stored in a ROM 402 or a computer program loaded into a RAM 403 from a storage unit 408. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An I / O interface 405 is also connected to the bus 404.

[0074] Various components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, such as a keyboard, a mouse, etc., an output unit 407, such as various types of displays, a speaker, etc., a storage unit 408, such as a magnetic disk, an optical disk, etc., and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0075] The computing unit 401 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs various methods and processes described above, such as the method log alert processing. For example, in some embodiments, the method log alert processing can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 400 via the ROM 802 and / or the communication unit 409. When the computer program is loaded onto the RAM 403 and executed by the computing unit 401, one or more steps of the method log alert processing described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the method log alert processing by any other appropriate means, such as by means of firmware.

[0076] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0077] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0080] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0081] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0082] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0083] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log alert processing method, characterized by, The method comprises: Collecting log data and performing data preprocessing to obtain a log to be detected; Performing feature detection on the log to be detected to obtain an abnormal vector, and performing retrieval in a vector database constructed in advance or in real time according to the abnormal vector to obtain a retrieval result; Accessing a general large model, generating a fault handling process according to the retrieval result, and pushing to an approval platform.

2. The log alert processing method of claim 1, wherein, Performing feature detection on the log to be detected to obtain an abnormal vector, comprising: Detecting the log to be detected by a predetermined algorithm to extract abnormal features, wherein the abnormal features include abnormal service names, abnormal times and abnormal status codes; Generating an abnormal vector according to the abnormal features and corresponding log texts.

3. The log alert processing method of claim 1, wherein, Retrieving in a vector database constructed in advance or in real time according to the abnormal vector to obtain a retrieval result, comprising: Retrieving in a vector knowledge base constructed in advance or in real time according to the abnormal vector to obtain fault handling information, wherein the fault handling information includes historical alarms, handling suggestions and manual feedbacks; The construction of the vector knowledge base comprises: Constructing a vector knowledge base from historical log alarm records and unstructured knowledge data, wherein the vector knowledge base contains historical alarms, handling suggestions and manual feedbacks; Vectorizing the data in the vector knowledge base to obtain a vector database.

4. The log alert processing method of claim 3, wherein, The vector knowledge base updating process comprises: Storing the fault handling process in a temporary cache area and storing by abnormal type; Performing semantic vectorization processing on the fault handling process in the temporary cache area by a timing task to obtain a vectorization processing result; Removing duplicates from the vectorization processing result by a cosine similarity algorithm to obtain a deduplicated vectorization result; Writing the deduplicated vectorization result into the vector knowledge base to update the vector knowledge base.

5. The log alert processing method of claim 3, wherein, Retrieving in a vector knowledge base constructed in advance or in real time according to the abnormal vector to obtain a retrieval result, comprising: Calculating the cosine similarity of the abnormal vector and the vectors in the vector knowledge base, and matching the most similar vector according to the cosine similarity result as the retrieval result; or Calculating the Euclidean distance of the abnormal vector and the vectors in the vector knowledge base, and matching the most similar historical knowledge vector according to the Euclidean distance result as the retrieval result.

6. The log alert processing method of claim 2, wherein, Performing feature detection on the log to be detected to obtain an abnormal vector, further comprising: Identifying the abnormal level of the abnormal type, calculating the abnormal threshold of the log to be detected by a sliding window algorithm according to the time series fluctuation range of the log to be detected; When the abnormality greater than the abnormal threshold is detected for a continuous preset number of times, the sensitivity of the abnormal threshold is adjusted according to the abnormal level to obtain the abnormal vector.

7. A log alert processing system characterized by, The system comprises: A data source module for collecting log data and performing data preprocessing to obtain a log to be detected; A data management module for performing feature detection on the log to be detected to obtain an abnormal vector, and performing retrieval in a vector database constructed in advance or in real time according to the abnormal vector to obtain a retrieval result; The large language model module is configured to access a general large model, generate a fault processing flow according to the search result, and push the fault processing flow to an approval platform.

8. The log alert processing system of claim 7, wherein, The data source module comprises a log collection unit and a data preprocessing unit, wherein, The log collection unit is configured to collect log information through a predetermined log framework to obtain to-be-processed logs. The data preprocessing unit is configured to perform format unification, structured processing and data cleaning on the to-be-processed logs to obtain to-be-detected logs.

9. The log alert processing system of claim 7, wherein, The data management module comprises a machine learning unit and a search enhancement generation unit, wherein, The machine learning unit is configured to perform feature detection on the to-be-detected logs to obtain an abnormal vector. The search enhancement generation unit is configured to perform search in a pre-constructed or real-time constructed vector database according to the abnormal vector to obtain a search result.

10. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

Images