Log processing method and device

Abstract

The invention discloses a log processing method, and the method can obtain an alarm result through processed system log data and related log data, matches an alarm rule based on semantic similarity, effectively reduces the false alarm and missing alarm, and improves the alarm accuracy. And a comprehensive and accurate alarm result is generated in combination with historical logs and context information, so that operation and maintenance personnel are assisted to quickly position faults. The log processing efficiency is greatly improved and the real-time requirement of large-scale data is met by utilizing efficient indexing and searching capability and by means of data vector rapid retrieval; an alarm rule and an embedded model are optimized according to user feedback, the manual maintenance workload is reduced, and the operation and maintenance cost is reduced; complex systems and variable service scenes can be processed, and different types of log data and alarm requirements can be met; different from completely depending on a large-scale pre-training model, the related log data is determined by using the data vector corresponding to the processed system log data, and the alarm rule is judged, so that computing resources are utilized more efficiently, and the system operation cost is reduced.

Description

Technical Field

[0001] This application relates to the technical field of network ranges, and in particular, to a log processing method and apparatus. Background Art

[0002] With the rapid development of information technology, the amount of log data generated by various systems and applications has increased explosively. The log data contains rich system operation information and abnormal situations. Effectively processing and alarming these log data is of great significance for ensuring the stable operation of the system, quickly locating faults, and optimizing operation and maintenance management. However, the existing log alarm processing methods mainly rely on rule matching and simple statistical analysis. These methods have the following limitations when dealing with large-scale and complex log data:

[0003] Limitations of rule matching: Fixed rules are difficult to cope with the diversity and dynamic variability of log data and cannot adapt to complex scenarios and rare faults.

[0004] Insufficient alarm accuracy: Simple statistical analysis methods are prone to false alarms and missed alarms, making it difficult for operation and maintenance personnel to accurately judge faults.

[0005] Low processing efficiency: Traditional methods are less efficient in processing massive log data and are difficult to meet real-time requirements.

[0006] Lack of context information: Existing methods usually only focus on individual log entries and lack comprehensive analysis of the entire system state and historical data, making it difficult to perform in-depth fault diagnosis.

[0007] High maintenance cost: With the development of business, the number and complexity of rules continue to increase, resulting in an increase in maintenance costs and making it difficult to achieve automation. Summary of the Invention

[0008] This application provides a log processing method and apparatus to achieve improved alarm accuracy, enhanced processing efficiency, reduced maintenance costs, flexible adaptation to complex scenarios, and optimized resource utilization.

[0009] In a first aspect, this application provides a log processing method, the method including:

[0010] Preprocessing the collected system log data to obtain processed system log data; and generating a data vector corresponding to each processed system log data;

[0011] For each data vector corresponding to each processed system log data, determining a plurality of relevant system log data with a similarity greater than a first preset threshold to the data vector corresponding to the processed system log data;

[0012] If the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, an alarm result is obtained according to the processed system log data and several related system log data of the processed system log data; wherein, the alarm result includes a result indicating whether an alarm needs to be triggered and alarm information;

[0013] If the alarm result includes a result indicating that an alarm needs to be triggered, the alarm information is sent to a user terminal.

[0014] In a second aspect, the present application provides a log processing device, and the device includes:

[0015] A first module, configured to preprocess the collected system log data to obtain processed system log data; and generate a data vector corresponding to each piece of the processed system log data respectively;

[0016] A second module, configured to, for the data vector corresponding to each piece of the processed system log data respectively, determine several related system log data whose similarity to the data vector corresponding to the processed system log data is greater than a first preset threshold according to the data vector corresponding to the processed system log data;

[0017] A third module, configured to, if the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, obtain an alarm result according to the processed system log data and several related system log data of the processed system log data; wherein, the alarm result includes a result indicating whether an alarm needs to be triggered and alarm information;

[0018] A fourth module, configured to, if the alarm result includes a result indicating that an alarm needs to be triggered, send the alarm information to a user terminal.

[0019] In a third aspect, the present application provides a readable medium, including execution instructions, and when a processor of an electronic device executes the execution instructions, the electronic device executes the method according to any one of the first aspect.

[0020] In a fourth aspect, the present application provides an electronic device, including a processor and a memory storing execution instructions, and when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of the first aspect.

[0021] It can be seen from the above technical solutions that the present application provides a log processing method, and the method can achieve the following beneficial effects:

[0022] By obtaining an alarm result based on the processed system log data and a number of relevant system log data of the processed system log data, it is possible to match alarm rules based on semantic similarity, effectively reduce false alarms and missed alarms, improve the accuracy of alarms, and, by combining historical log data and context information, generate more comprehensive and accurate alarm results to help operation and maintenance personnel quickly locate faults. Additionally, by utilizing efficient indexing and search capabilities and combining with the fast retrieval of data vector search, the processing efficiency of log data is significantly improved to meet the real-time requirements of large-scale data. Moreover, by optimizing alarm rules and embedding models through user feedback, the workload of manually maintaining rules is reduced and the operation and maintenance costs are lowered. It can be seen that the method provided in this application can handle complex systems and changing business scenarios, adapt to different types of log data and alarm requirements, and, compared with relying entirely on large-scale pre-trained models, using the data vectors corresponding to the processed system log data respectively to determine a number of relevant system log data and judge whether it meets the alarm rules can utilize computing resources more efficiently and reduce the system operation cost.

[0023] The further effects of the above non-conventional preferred methods will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] To more clearly illustrate the embodiments of the present application or the existing technical solutions, the following will briefly introduce the drawings required for use in the description of the embodiments or the existing technical solutions. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0025] Figure 1 It is a schematic flowchart of a log processing method provided by the present application;

[0026] Figure 2 It is a schematic structural diagram of a log processing device provided by the present application;

[0027] Figure 3 It is a schematic structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To make the objectives, technical solutions, and advantages of the present application clearer, the following will clearly and completely describe the technical solutions of the present application in conjunction with specific embodiments and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0029] The following will describe various non - restrictive embodiments of the present application in detail with reference to the accompanying drawings.

[0030] Refer to Figure 1 , which shows a log processing method in an embodiment of the present application. In this embodiment, the method may, for example, include the following steps:

[0031] S101: Pre - process the collected system log data to obtain the pre - processed system log data; and, generate a data vector corresponding to each piece of the pre - processed system log data respectively.

[0032] In this embodiment, the pre - processing includes at least one of the following: cleaning process, deduplication process, formatting process. As an example, a Logstash configuration file can be used to define the collection path and transmission method of the system log data, transmit the collected log data to a preset database (such as Elasticsearch) through the HTTP protocol, and then, the Ingest Pipeline function of the preset database (such as Elasticsearch) can be used to define a pre - processing pipeline, including pre - processing operations such as cleaning, deduplication, and formatting. Then, pre - process the collected system log data to obtain the pre - processed system log data. For example, clean the information at the info and debug levels; remove useless fields, such as removing: "_type": "_doc", which identifies the elasticseach index type; remove the same error logs with the same timestamp.

[0033] Next, a data vector corresponding to each piece of the pre - processed system log data can be generated. As an example, generate a corresponding index for each piece of the pre - processed system log data in the preset database, and store the pre - processed system log data in the corresponding index. Specifically, first create a corresponding index for each piece of the pre - processed system log data in the preset database. For example, an index can be created in Elasticsearch. Then, create a corresponding index definition mapping (mappings) for the pre - processed system log data, where the index mapping includes the types and attributes of the log fields of the pre - processed system log data, such as fields like log time, log level, log content, etc. Store the pre - processed system log data in the corresponding index, that is, this index is used to store the pre - processed system log data.

[0034] Subsequently, a preset embedding model can be utilized to generate data vectors respectively corresponding to each processed system log data. The preset embedding model is the E5 multilingual ML model. That is, the built-in embedding model of Elasticsearch (such as the E5 multilingual ML model) can be used to convert the log content of the system log data into a vector representation. Specifically, by interacting with the serverless deployed Elasticsearch service, a new text embedding inference service is created; first, a client connected to Elasticsearch is set up, and the necessary authentication information and timeout settings are specified; then, the specific configuration of the inference service is defined, including the model and service settings used; finally, the inference service is created by calling the put method, enabling it to handle text embedding tasks (that is, converting the log content of the system log data into a vector representation, which means generating data vectors respectively corresponding to each processed system log data). This approach is very suitable for scenarios that require quick startup and shutdown, on-demand scaling, and high cost-effectiveness, especially when dealing with natural language processing tasks such as semantic search and text classification.

[0035] S102: For each data vector respectively corresponding to each processed system log data, determine a number of relevant system log data whose similarity to the processed system log data is greater than a first preset threshold according to the data vector corresponding to the processed system log data.

[0036] In this embodiment, several relevant system log data with a similarity greater than a first preset threshold to the processed system log data can be determined according to the data vector corresponding to the processed system log data. As an example, the K-nearest neighbor algorithm can be used to query all the processed system log data in the preset database based on the data vector corresponding to the processed system log data, so as to obtain several processed system log data with a similarity greater than the first preset threshold to the processed system log data, and use the several processed system log data with a similarity greater than the first preset threshold to the processed system log data as the relevant system log data. For example, the search function of Elasticsearch can be used in combination with vector search and text search, and traditional searches (such as keyword matching and time range filtering) can be combined with semantic similarity searches. Specifically, the vector search ability of Elasticsearch needs to be used to perform a KNN (k-Nearest Neighbors) query on the embedding vectors of the log content, so as to find several processed system log data with a similarity greater than the first preset threshold to the processed system log data. For instance, the processed system log data has been converted into a vector representation (i.e., data vector). Next, a composite query can be constructed, which includes both conventional boolean conditions and a vector search part, so as to query all the processed system log data in the preset database based on the data vector corresponding to the processed system log data, and obtain several processed system log data with a similarity greater than the first preset threshold to the processed system log data. It can be understood that for each processed system log data, its text embedding vector (i.e., data vector) is first calculated, and then the KNN query of Elasticsearch is used to retrieve the most similar several records (i.e., several relevant system log data with a similarity greater than the first preset threshold to the processed system log data) from the historical log data (i.e., the preset database).

[0037] S103: If the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, an alarm result is obtained according to the processed system log data and several relevant system log data of the processed system log data.

[0038] Among them, the alarm result includes the result of whether an alarm needs to be triggered and the alarm information.

[0039] In this embodiment, an alarm rule based on semantic similarity can be used to determine whether alarm needs to be triggered for system log data. Specifically, when a newly received log entry is highly semantically similar to the system log data of severe faults or abnormal situations in history, an alarm is triggered. Therefore, if the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, an alarm result can be obtained based on the processed system log data and several related system log data of the processed system log data.

[0040] Specifically, the processed system log data and several related system log data of the processed system log data can be input into a large language model, and the large language model is used to determine whether to trigger an alarm result and initial alarm information based on the context of the processed system log data and several related system log data of the processed system log data. That is to say, the processed system log data and several related system log data of the processed system log data can be input into a large language model (LLM), and the LLM is allowed to judge whether to trigger an alarm and generate detailed alarm information according to the context (that is, to determine whether to trigger an alarm result and initial alarm information).

[0041] The initial alarm information is input into a Retrieval-Augmented Generation (RAG) model to obtain alarm information, where the alarm information includes an alarm level, alarm content, and alarm time. Moreover, the result of whether to trigger an alarm and the alarm information are used as an alarm result.

[0042] S104: If the alarm result includes a result that an alarm needs to be triggered, the alarm information is sent to the user terminal.

[0043] In this embodiment, if the alarm result includes a result that an alarm needs to be triggered, the alarm information is sent to the user terminal through an alarm notification system (such as Email, SMS, Slack, etc.).

[0044] In one implementation manner of this embodiment, the method further includes:

[0045] In response to the feedback information of the user on the alarm information, the large language model and / or the retrieval-augmented generation model are adjusted.

[0046] That is to say, this embodiment can collect feedback information for the alarm information and optimize the large language model and / or the retrieval-augmented generation model through the feedback information. Next, taking a user interaction example, the user is asked about the validity of the alarm, and whether to update the alarm rule or the embedding model is determined according to the user's input: 1. User interaction: The program will first pause and wait for the user to input. The user needs to answer whether the alarm is valid and respond by inputting "yes" or "no". 2. Conditional branch: According to the user's input, the program will enter different processing paths: - If the user inputs "no" (case-insensitive), it is considered that the alarm is invalid, and the program will try to perform some update operations; if the user inputs "yes" or any other value, the program will not change the alarm rule or the model (i.e., the large language model and / or the retrieval-augmented generation model). 3. Update mechanism: When the user confirms that the alarm information is invalid, theoretically, specific code should be included here to adjust the parameters of the alarm system, improve the detection algorithm, or retrain the prediction model (i.e., improve or retrain the large language model and / or the retrieval-augmented generation model) to improve the accuracy of the model.

[0047] As can be seen from the above technical solution, the present application provides a log processing method, and the method can achieve the following beneficial effects:

[0048] By obtaining the alarm result based on the processed system log data and several related system log data of the processed system log data, it is possible to perform alarm rule matching based on semantic similarity, effectively reduce false alarms and missed alarms, improve the accuracy of alarms, and, be able to combine historical log data and context information to generate more comprehensive and accurate alarm results to help the operation and maintenance personnel quickly locate faults. Also, by using efficient indexing and search capabilities and combining the fast retrieval of data vector search, the processing efficiency of log data is significantly improved, meeting the real-time requirements of large-scale data. And, by optimizing the alarm rule and the embedding model through user feedback, the workload of manual rule maintenance is reduced, and the operation and maintenance cost is lowered. It can be seen that the method provided by the present application can handle complex systems and changing business scenarios, adapt to different types of log data and alarm requirements, and, compared with relying entirely on large-scale pre-trained models, using the data vectors corresponding to the processed system log data respectively to determine several related system log data and judge whether it meets the alarm rule can more efficiently utilize computing resources and reduce the system operation cost. That is to say, the present application collects, preprocesses, indexes, vectorizes, matches alarm rules, generates alarms, and provides alarm feedback for the system log data. By combining vector search and RAG summary and search, the accuracy and processing efficiency of log alarms are improved.

[0049] That is to say, the present application has the following remarkable beneficial effects:

[0050] Improve alarm accuracy: Through vector search and RAG technology, it is possible to match alarm rules based on semantic similarity, effectively reducing false alarms and missed alarms, and improving the accuracy of alarms.

[0051] Enhance processing efficiency: Utilize the efficient indexing and search capabilities of Elasticsearch, combined with the fast retrieval of vector search, to significantly enhance the processing efficiency of log data and meet the real-time requirements of large-scale data.

[0052] Enhance context understanding: RAG technology can combine historical log data and context information to generate more comprehensive and accurate alarm information, helping operation and maintenance personnel quickly locate faults.

[0053] Reduce maintenance costs: Optimize alarm rules and embedding models through user feedback, reducing the workload of manual rule maintenance and lowering operation and maintenance costs.

[0054] Flexibly adapt to complex scenarios: The present invention can handle complex systems and changing business scenarios, adapting to different types of log data and alarm requirements.

[0055] Optimize resource utilization: Compared with relying entirely on large-scale pre-trained models, the RAG technology combined with vector search can utilize computing resources more efficiently and reduce system operation costs.

[0056] As Figure 2 shown, it is a specific embodiment of a log processing device described in this application. The device in this embodiment is the entity device for executing the method described in the above embodiment. Its technical solution is essentially the same as that of the above embodiment, and the corresponding descriptions in the above embodiment also apply to this embodiment. The device in this embodiment is applied to a network range platform, and the device includes:

[0057] The first module 201 is used to preprocess the collected system log data to obtain the processed system log data; and generate data vectors respectively corresponding to each piece of the processed system log data;

[0058] The second module 202 is used to determine a number of relevant system log data with a similarity greater than a first preset threshold to the processed system log data according to the data vectors respectively corresponding to each piece of the processed system log data;

[0059] The third module 203 is configured to, if the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, obtain an alarm result according to the processed system log data and a number of related system log data of the processed system log data; wherein, the alarm result includes a result of whether to trigger an alarm and alarm information;

[0060] The fourth module 204 is configured to, if the alarm result includes a result of needing to trigger an alarm, send the alarm information to a user terminal.

[0061] Optionally, the preprocessing includes at least one of the following: cleaning processing, duplicate removal processing, and formatting processing.

[0062] Optionally, the first module 201 is configured to:

[0063] Generate a corresponding index for each processed system log data in a preset database, and store the processed system log data in the corresponding index;

[0064] Use a preset embedding model to generate a data vector corresponding to each processed system log data respectively.

[0065] Optionally, the first module 201 is configured to:

[0066] Create a corresponding index for each processed system log data in a preset database;

[0067] Create a corresponding index definition mapping for the processed system log data, where the index mapping includes the types and attributes of the log fields of the processed system log data;

[0068] Store the processed system log data in the corresponding index.

[0069] Optionally, the preset embedding model is the E5 multilingual ML model.

[0070] Optionally, the second module 202 is configured to:

[0071] Adopt the K-nearest neighbor algorithm to query all the processed system log data in the preset database based on the data vector corresponding to the processed system log data, obtain a number of processed system log data whose similarity to the processed system log data is greater than a first preset threshold, and use the number of processed system log data whose similarity to the processed system log data is greater than the first preset threshold as related system log data.

[0072] Optionally, the third module 203 is configured to:

[0073] Input the processed system log data and several related system log data of the processed system log data into a large language model, and use the large language model to determine the result of whether an alarm needs to be triggered and the initial alarm information according to the context of the processed system log data and several related system log data of the processed system log data;

[0074] Input the initial alarm information into a retrieval-augmented generation model to obtain alarm information, where the alarm information includes an alarm level, alarm content, and alarm time;

[0075] Take the result of whether an alarm needs to be triggered and the alarm information as the alarm result.

[0076] Optionally, the fourth module 204 is used for:

[0077] If the alarm result includes a result that an alarm needs to be triggered, send the alarm information to the user terminal through an alarm notification system.

[0078] Optionally, the device further includes a fifth module for:

[0079] In response to the feedback information of the user regarding the alarm information, adjust the large language model and / or the retrieval-augmented generation model.

[0080] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0081] The processor, network interface, and memory can be interconnected through an internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 3It is represented only by a bidirectional arrow, but it does not mean that there is only one bus or one type of bus.

[0082] A memory for storing executable instructions. Specifically, the executable instructions are computer programs that can be executed. The memory may include a memory and a non-volatile memory, and provide the executable instructions and data to the processor.

[0083] In a possible implementation manner, the processor reads the corresponding executable instructions from the non-volatile memory into the memory and then runs them, or can also obtain the corresponding executable instructions from other devices to form a log processing device at the logical level. The processor executes the executable instructions stored in the memory to implement the log processing method provided in any embodiment of the present application through the executed executable instructions.

[0084] The above as in the present application Figure 1 The method executed by the log processing device provided in the embodiments shown in the present application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0085] The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being completed by a hardware decoding processor, or completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0086] An embodiment of the present application also provides a readable medium. The readable storage medium stores execution instructions. When the stored execution instructions are executed by a processor of an electronic device, the electronic device can execute the log processing method provided in any embodiment of the present application, and is specifically used to execute the above-mentioned log processing method.

[0087] The electronic device described in each of the foregoing embodiments may be a computer.

[0088] Those skilled in the art should understand that the embodiments of the present application may be provided as a method or a computer program product. Therefore, the present application may adopt a form of all hardware embodiments, all software embodiments, or a combination of software and hardware.

[0089] The embodiments in the present application are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the description of the method embodiments.

[0090] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity or device. Without more limitations, the element defined by the statement "including one..." does not exclude the existence of other identical elements in the process, method, commodity or device including the said element.

[0091] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A log processing method, characterized in that, The method includes: Preprocessing the collected system log data to obtain processed system log data; and generating data vectors respectively corresponding to each piece of processed system log data; For each data vector respectively corresponding to each piece of processed system log data, determining a number of relevant system log data with a similarity greater than a first preset threshold to the processed system log data according to the data vector corresponding to the processed system log data; If the similarity between the data vector corresponding to a piece of processed system log data and a preset alarm rule vector meets a second preset threshold, then obtaining an alarm result according to the processed system log data and the number of relevant system log data of the processed system log data; wherein, the alarm result includes a result of whether to trigger an alarm and alarm information; If the alarm result includes a result of needing to trigger an alarm, then sending the alarm information to a user terminal.

2. The method according to claim 1, wherein The preprocessing includes at least one of the following: cleaning processing, duplicate removal processing, formatting processing.

3. The method according to claim 1, characterized in that The generating data vectors respectively corresponding to each piece of processed system log data includes: Generating a corresponding index for each piece of processed system log data in a preset database, and storing the processed system log data in the corresponding index; Using a preset embedding model to generate data vectors respectively corresponding to each piece of processed system log data.

4. The method according to claim 3, wherein The generating a corresponding index for each piece of processed system log data in a preset database and storing the processed system log data in the corresponding index includes: Creating a corresponding index for each piece of processed system log data in a preset database; Creating an index mapping for the corresponding index created for the processed system log data, wherein the index mapping includes the types and attributes of the log fields of the processed system log data; Storing the processed system log data in the corresponding index.

5. The method according to claim 3, wherein The preset embedding model is an E5 multilingual ML model.

6. The method according to claim 3, characterized in that, The determining a number of relevant system log data with a similarity greater than a first preset threshold to the processed system log data according to the data vector corresponding to the processed system log data includes: Adopting a K-nearest neighbor algorithm to query all the processed system log data in the preset database based on the data vector corresponding to the processed system log data, obtaining a number of processed system log data with a similarity greater than the first preset threshold to the processed system log data, and using the number of processed system log data with a similarity greater than the first preset threshold to the processed system log data as relevant system log data.

7. The method according to claim 1, characterized in that, The obtaining an alarm result according to the processed system log data and the number of relevant system log data of the processed system log data includes: Input the processed system log data and several related system log data of the processed system log data into a large language model, and use the large language model to determine the result of whether an alarm needs to be triggered and the initial alarm information according to the context of the processed system log data and several related system log data of the processed system log data; Input the initial alarm information into a retrieval-augmented generation model to obtain alarm information, where the alarm information includes an alarm level, alarm content, and alarm time; Use the result of whether an alarm needs to be triggered and the alarm information as the alarm result.

8. The method according to claim 1, characterized in that, If the alarm result includes a result that an alarm needs to be triggered, then send the alarm information to the user terminal, including: If the alarm result includes a result that an alarm needs to be triggered, send the alarm information to the user terminal through an alarm notification system.

9. The method according to claim 7, characterized in that The method further includes: In response to the feedback information of the user regarding the alarm information, adjust the large language model and / or the retrieval-augmented generation model.

10. A log processing device, characterized in that, The device includes: A first module for preprocessing the collected system log data to obtain processed system log data; and generating a data vector corresponding to each processed system log data; A second module for, for each data vector corresponding to each processed system log data, determining several related system log data with a similarity greater than a first preset threshold to the data vector corresponding to the processed system log data according to the data vector corresponding to the processed system log data; A third module for, if the similarity between the data vector corresponding to a processed system log data and a preset alarm rule vector meets a second preset threshold, obtaining an alarm result according to the processed system log data and several related system log data of the processed system log data; where the alarm result includes a result of whether an alarm needs to be triggered and alarm information; A fourth module for, if the alarm result includes a result that an alarm needs to be triggered, sending the alarm information to the user terminal.

Images