Log alarm method and device, electronic equipment, storage medium and program

Abstract

The embodiment of the invention discloses a log alarm method and device, electronic equipment, a storage medium and a program, and the method comprises the steps: carrying out the dynamic partitioning of an original collection log according to a log type, and obtaining a partitioned log; performing unified processing on the data structure of each type of partition log through a large model to obtain a structured partition log; key log events of the structured partition logs are automatically recognized through the large model, and a key log event list is obtained; and monitoring the original collection log in real time according to the key log event list, and generating real-time log alarm information. According to the technical scheme provided by the embodiment of the invention, the log alarm processing efficiency, the automation degree and the self-adaptability can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of log data security technology, and in particular to a log alarm method, device, electronic device, storage medium and program. Background Technology

[0002] In today's modern distributed IT (Information Technology) infrastructure and cloud environments, massive amounts of log data are core data for system operation and maintenance, security monitoring, and fault diagnosis. Logs include system event records such as errors, warnings, and access logs, as well as application behavior tracking such as API (Application Programming Interface) calls and network traffic information, generating a large amount of data daily. These logs need to be ingested, parsed, and analyzed in real time to identify abnormal events and trigger alarms to support root cause analysis. With the popularization of AI (Artificial Intelligence) technology, log systems emphasize automation and adaptability to cope with dynamic threats and complex environments.

[0003] Currently, log systems typically use static rules to create periodic anomaly detection jobs and employ anomaly detection algorithms (such as implementing data ingestion anomalies) or time series analysis (such as analyzing network traffic anomalies) to detect outliers and generate alarm dashboards for log data.

[0004] In the process of developing this invention, the inventors discovered that while existing technologies are relatively mature, they have significant limitations, particularly in terms of automation, adaptability, and efficiency. The main problems are as follows: Existing log detection jobs rely heavily on manual configuration, and parsing or alarm rules require manual writing by experts. Log systems use static rules for anomaly detection, making them difficult to adapt to, and requiring manual intervention to cope with sudden events. The interpretability and interactivity of the entire log data detection process are weak. Summary of the Invention

[0005] This invention provides a log alarm method, apparatus, electronic device, storage medium, and program that can improve the efficiency, automation, and adaptability of log alarm processing.

[0006] According to one aspect of the present invention, a log alarm method is provided, comprising:

[0007] The original collected logs are dynamically partitioned according to the log type to obtain partitioned logs;

[0008] By unifying the data structure of each type of partition log through a large model, structured partition logs are obtained.

[0009] The large model is used to automatically identify key log events of the structured partition log, and a key log event list is obtained.

[0010] Real-time log alarm information is generated by monitoring the original collected log according to the key log event list.

[0011] According to another aspect of the present application, a log alarm device is provided, comprising:

[0012] A log partition module is configured to dynamically partition original collected logs according to log types, and obtain partition logs.

[0013] A partition log structured processing module is configured to uniformly process the data structure of each type of partition log by a large model, and obtain structured partition logs.

[0014] A key log event list generation module is configured to automatically identify key log events of the structured partition log by the large model, and obtain a key log event list.

[0015] A log alarm information generation module is configured to monitor the original collected log in real time according to the key log event list, and generate real-time log alarm information.

[0016] According to another aspect of the present application, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory in communication connection with the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the log alarm method according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the log alarm method according to any one of the embodiments of the present application.

[0021] According to another aspect of the present application, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the log alarm method according to any one of the embodiments of the present application.

[0022] The embodiment of the application obtains the partitioned log by dynamically partitioning the original collected log according to the log type, and obtains the structured partitioned log by uniformly processing the data structure of each type of partitioned log by the large model. Further, the key log event list is obtained by automatically identifying the key log event of the structured partitioned log by the large model, and the real-time log alarm information is generated by monitoring the original collected log in real time according to the key log event list, so as to solve the problems of low efficiency, automation degree and self-adaptability of the existing log alarm processing process, and improve the efficiency, automation degree and self-adaptability of the log alarm processing.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of a log alarm method provided by the first embodiment of the application;

[0026] Figure 2 is a flowchart of a log alarm method provided by the second embodiment of the application;

[0027] Figure 3 is a schematic diagram of a log alarm device provided by the third embodiment of the application;

[0028] Figure 4 is a structural schematic diagram of an electronic device provided by the fourth embodiment of the application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the application.

[0030] It is to be understood that the terms "including", "comprising", "having" and "with" are open-ended and are used as equivalents that do not exclude additional steps or elements. It is also to be understood that, unless otherwise indicated herein, the singular forms of the terms "a", "an" and "the" include the plural forms of the terms.

[0031] Embodiment one

[0032] Figure 1 is a flowchart of a log alarm method provided by Embodiment One of the present application. The present embodiment can be applied to the case of using a large model to structure log data and generate a log event list to assist in alarm analysis. The method can be executed by a log alarm device, which can be implemented in software and / or hardware and generally integrated in an electronic device, which can be a terminal device or a server device, as long as it can execute the log alarm method. The present application does not limit the specific type of electronic device. Correspondingly, as shown in Figure 1 the method includes the following operations:

[0033] S110, dynamically partitioning the original collected log according to the log type to obtain partitioned log.

[0034] The original collected log can be a log originally collected and obtained by the log management system. The partitioned log can be each type of log obtained by the log management system after classifying and processing the original collected log.

[0035] In the present embodiment, the log management system can interface with various system platforms or applications through an interface to collect and import the original collected log. Since the log management system can comprehensively manage the logs generated by different types of systems, platforms or applications, after obtaining the original collected log, the log management system can first analyze the log type of the original collected log, and then dynamically partition the original collected log based on the log type. Dynamic partitioning means that log data of the same log type is divided into a type of partitioned log. Each type of partitioned log represents a log type. Each type of partitioned log has the same or similar log structure.

[0036] S120, uniformly processing the data structure of each type of partitioned log through a large model to obtain structured partitioned log.

[0037] The structured partitioned log can be log data obtained by structuring the partitioned log.

[0038] The large model used in the embodiments of the present application can be any available large language model (LLM). The large language model is also called a large-scale language model, which refers to a deep learning model trained using a large amount of relevant data (such as text data, voice data, or combination of text and image data) to process a certain sequence (such as text, voice, image, or audio and video, and combination of text and image sequence types). The model can generate natural language text or understand the meaning of language text. Such a model usually has a parameter size of more than several billion. The large language model can process various natural language tasks, such as text classification, question answering, and dialogue, and is widely used. The input of the large language model is data, which can be text data, voice data, or combination of text and image data, etc. The large language model can obtain the corresponding word vector representation by encoding the input data, and further decode the encoded word vector to automatically complete the processing of the input data and obtain the output data corresponding to the input data. For example, the large language model can input text, process and predict the input text, and output the corresponding response text.

[0039] Since each type of partitioned log has the same or similar data structure, in order to improve the efficiency of log data processing, a large model can be introduced to uniformly process the unstructured or semi-structured data structure of each type of partitioned log, so that each type of partitioned log has the same data structure. Correspondingly, each type of partitioned log can be uniformly processed to obtain a corresponding structured partitioned log. The data structure of the structured partitioned log generated by processing different types of partitioned logs can be different.

[0040] S130, automatically identifying key log events of the structured partitioned log by the large model to obtain a key log event list.

[0041] The key log event can be a log event recorded by the system during operation, which details the operation, time, location, and participating entities.

[0042] The log data usually records key log events. In order to further improve the efficiency of log data processing, the large model can be used to automatically identify the key log events included in the structured partitioned log, and the large model can be used to automatically arrange and output the key log event list corresponding to the structured partitioned log. For example, the type of key log event included in the key log event list can be a security-related event, a system operation event, a business operation event, and an abnormal behavior, etc. The key log event list can be used as a basis for security monitoring, event response, and compliance audit, and can be used for subsequent log alarm processing.

[0043] S140, monitoring the original collection log in real time according to the key log event list to generate real-time log alarm information.

[0044] Correspondingly, after obtaining the key log event list, the key log event list can be used as a reference to monitor the original collection log in real time to detect whether a key alarm event appears in the original collection log, and real-time log alarm information is generated for rapid alarm after determining that an alarm event appears in the original collection log.

[0045] Therefore, the above technical solution automatically parses each type of partitioned log by using a large model to convert unstructured or semi-structured partitioned logs into a unified structured format, and automatically arranges and outputs a key log event list required for log monitoring by using a large model, which greatly improves the automation degree of the log alarm processing process, the log data processing efficiency, and the adaptability of the alarm performance.

[0046] According to the embodiment of the application, after the original collection log is dynamically partitioned according to the log type to obtain the partitioned log, the data structure of each type of partitioned log is uniformly processed by using a large model to obtain a structured partitioned log. Further, the key log event list is obtained by automatically identifying the key log event of the structured partitioned log by using a large model, and the original collection log is monitored in real time according to the key log event list to generate real-time log alarm information, thereby solving the problems of low efficiency, low automation degree, and low adaptability of the existing log alarm processing process, and improving the efficiency, automation degree, and adaptability of the log alarm processing.

[0047] Embodiment Two

[0048] Figure 2 is a flowchart of a log alarm method provided by the second embodiment of the application. The second embodiment is based on the above-mentioned embodiments and is specific. In the second embodiment, various specific and optional implementation manners of dynamically partitioning the original collection log according to the log type, uniformly processing the data structure of each type of partitioned log, and automatically identifying the key log event of the structured partitioned log are given. Correspondingly, as shown in Figure 2 the method of the second embodiment can include:

[0049] S210, identifying the log stream source of the original collection log, dynamically partitioning the original collection log according to the log stream source of the original collection log to obtain the partitioned log.

[0050] In the embodiments of the present application, when dynamically partitioning the original collection logs, the original collection logs can be classified into one type of partitioned logs by identifying the log stream sources of the original collection logs and taking the log stream sources as the basis. For example, the log stream sources can include, but are not limited to, IP, host, application, etc. For example, the original collection logs of the same IP can be classified into one type of partitioned logs, or the original collection logs generated by the same host can be classified into one type of partitioned logs, etc.

[0051] S220, collecting partitioned log samples from each of the partitioned logs, generating a structured prompt according to the partitioned log samples and a structure data processing template, inputting the structured prompt into the large model, and processing the data structure of the partitioned log samples according to the structured prompt by the large model to obtain a partitioned structured field.

[0052] The partitioned log samples can be several log samples in the partitioned logs. The structured prompt can be a prompt input to the large model, which is used to prompt the large model to uniformly process the data structure of the partitioned logs. The partitioned structured field can be a field required for the structured processing of the data structure of the partitioned logs. The structure data processing template can be a template for indicating the generation of a standard data structure.

[0053] Since there can be a large amount of log data in each type of partitioned log, if all the log data of a type of partitioned log is processed by the large model, the storage and reading efficiency of the massive data directly affects the processing speed, and thus the processing efficiency of the large model is significantly reduced. To solve the above technical problem, several partitioned log samples can be collected from each type of partitioned log. For example, 3-5 logs of one type of partitioned log can be collected as the partitioned log sample corresponding to the partitioned log.

[0054] The large model can use a general structured prompt template as a starting point, which is based on common log standards such as syslog (system log) or apache common log format (Apache HTTP server record access log standard text format), etc. For example, the general structured prompt template can be {“timestamp”:“str”,”level”:”str”,“source”:”str”,”emssage”:”str”,”entities”:”dict”}. However, the general structured prompt template is not fixed, and the large model can dynamically expand or adjust the related fields of the general structured prompt template according to the input log sample.

[0055] After collecting the partition log samples from the partition logs, the content of the partition log samples can be further filled into the preset structured data processing template to generate a structured prompt, and the generated structured prompt is input into the large model, and the large model is used to uniformly process and preliminarily evaluate the data structure of the partition log sample according to the structured prompt, to obtain a partition structured field.

[0056] In an optional embodiment of the present application, the generation of the structured prompt according to the partition log sample and the structured data processing template can include: determining the extended field adapted to the partition log sample according to the type of the partition log sample; generating an extended field example according to the extended field adapted to the partition log sample; and generating the structured prompt according to the partition log sample, the structured data processing template and the extended field example.

[0057] It can be understood that part of the partition log can include other extended fields in addition to the general field. Therefore, when generating the structured prompt, the type of the partition log sample can be analyzed to determine the extended field adapted to each type of partition log sample. It can be understood that the extended field adapted to different types of partition log samples can also be different. Further, the extended field example can be generated according to the extended field adapted to the partition log sample, so that the generated extended field example is added to the structured data processing template to generate the structured prompt in combination with the partition log sample.

[0058] In a specific example, the structured prompt input into the large model can be as follows:

[0059] The following partition log samples are analyzed to infer common fields and types. Based on the general template {general schema}, a targeted schema is generated. Partition log samples:

[0060] Log1: 2025-11-03 10:00:15 ERROR / api / user 500 192.168.1.10 Connectiontimeout

[0061] Log2: 2025-11-03 10:10:25 ERROR / db / query 500 192.168.1.20 Databaseconnection failed

[0062] Output: Extended JSON schema, for example, add "endpoint" field.

[0063] In the above structured prompt, "schema" represents a field.

[0064] S230, extracting the field values of the partition structured field from the original collection log by a named entity recognition tool, and automatically adding to the partition structured field to obtain the structured partition log.

[0065] The named entity recognition tool can identify and extract entity fields from data.

[0066] After determining the partition structured field in the data structure, the partition structured field can be used as a standard for the data structure, and the named entity recognition tool can be used to extract the field values corresponding to each partition structured field from the original collection log and automatically add them to the partition structured field, thereby obtaining the structured partition log corresponding to each partition log data.

[0067] To further improve the efficiency of structured processing, the log management system can also maintain a knowledge base of partition structured fields for storing various processed partition structured fields. Before uniform processing of the data structure of each type of partition log by the large model, the knowledge base of partition structured fields can be queried first to determine whether the knowledge base of partition structured fields stores the partition structured field adapted to the current partition log. If it is determined that the knowledge base of partition structured fields stores the partition structured field adapted to the current partition log, the data structure of the current partition log can be directly processed by the large model based on the partition structured field queried from the knowledge base.

[0068] The data structure of the current partition log is uniformly processed by the large model to ensure that the structured partition log output by the current partition log is consistent.

[0069] S240, generating an initial key event list of the structured partition log, and mining log patterns of the structured partition log.

[0070] The initial key event list can be an initial key event list constructed according to an empirical value.

[0071] In an embodiment of the present application, the initial key event list of the structured partition log can be generated based on user domain knowledge. For example, the key event list can match high-risk words with high message severity in level / message / entities, such as ["ERROR", "FATAL", "500", "timeout", "failed", "unauthorized", "intrusion"], case-insensitive regular expressions, etc.

[0072] Meanwhile, a pattern mining algorithm such as drain (An Online Log Parsing Approach with Fixed Depth Tree) can be used to mine the patterns of the logs corresponding to the structured partitioned logs, to determine the log patterns of each structured partitioned log. The log pattern refers to the configuration mode used to control the log recording behavior in a system or application program, which determines the generation, storage, archiving and usage strategy of the log. For example, in a database system, the log pattern mainly involves the log recording level and archiving strategy; the log pattern in an application framework mainly focuses on the log level and output format.

[0073] S250, generating an event extraction prompt according to the log pattern of the structured partitioned log, and inputting the event extraction prompt into the large model to extract a new event list from the log pattern of the structured partitioned log according to the event extraction prompt.

[0074] S260, updating the initial key event list according to the new event list to obtain the key log event list.

[0075] The event extraction prompt can be a prompt input to the large model, which is used to prompt the large model to extract new events from the log pattern of the structured partitioned log and update the event list.

[0076] When updating the event list, the event extraction prompt can be generated according to the mined log pattern of the structured partitioned log, and the generated event extraction prompt is input into the large model, so that the large model further mines and extracts new key events from the log pattern of the structured partitioned log according to the event extraction prompt, and generates a new event list based on the extracted new key events. Further, the new event list is added to the initial key event list to update the initial key event list, to obtain the key log event list.

[0077] In a specific example, the event extraction prompt can include the following content:

[0078] Analyze these log patterns, extract potential new key events (high-risk words, such as abnormal descriptions). Ignore common words. Output: new event list + reason.

[0079] It should be noted that the list of key log events is not static and can be dynamically extended by the large model. Ensure that the log management system can adapt to new log patterns, such as the emerging error "auth_bypass" (authentication bypass), to avoid outdated rules. When a new log pattern appears, the list of key log events can be evaluated and updated. The new log pattern currently appears, or user feedback logs, are submitted to the large model for prompt evaluation, and the new event list and event extraction reasons output by the large model are analyzed. Thus, according to the event extraction reasons, some or all of the new events in the new event list are added to the initial key event list. Alternatively, the large model can also output a complete list of key log events based on the log pattern, that is, the large model automatically updates the initial key event list based on the new event list to obtain the list of key log events. The embodiments of the present application do not limit the processing method of the list of key log events.

[0080] S270, real-time monitoring of the original collected log according to the list of key log events to generate real-time log alarm information.

[0081] It can be understood that the list of key log events only lists the key information that may appear abnormal or malfunction in the log, but does not include complete alarm information. In order to realize real-time monitoring of log data, a few-shot prompt template (a prompt engineering technology) can be used to generate an SPL (Search Processing Language, a language for querying and processing log data) query statement for querying and monitoring the original collected log according to the list of key log events. Optionally, the rationality and compliance of the SPL query statement can be verified, and after passing the verification, it is saved as a search. The cron integrated scheduler (a mechanism that uses cron as the core component to manage and execute scheduled tasks) queries the original collected log of the log management system in real time or periodically according to the SPL query statement, and generates real-time log alarm information when it is determined that the key word in the original collected log matches the key event in the list of key log events.

[0082] Optionally, the log management system can query the full amount of original collected log through the SPL query statement, or can only monitor the incremental original collected log by setting the way of burying points or tags. That is, the original collected log that has been queried will not be queried repeatedly, thereby improving the efficiency of log alarm processing.

[0083] In an optional embodiment of the present application, after the real-time log alarm information is generated, the method further comprises: extracting an alarm keyword of the real-time log alarm information, retrieving a target log chain according to the alarm keyword, and generating a multi-dimensional comprehensive analysis report of the real-time log alarm information according to the target log chain.

[0084] The alarm keyword can be a keyword extracted from the log alarm information and can represent key information of the log alarm. The target log chain can be a log chain including the alarm keyword.

[0085] After the log alarm is triggered, the log management system can extract an alarm keyword of the real-time log alarm information, determine a target log chain matched with the alarm keyword, retrieve the target log chain according to the alarm keyword to obtain more detailed information related to the alarm, or submit the target log chain to a large model for interpretation to give log interpretation analysis and related solution suggestions, so as to generate a multi-dimensional comprehensive analysis report of the real-time log alarm information according to the target log chain. The multi-dimensional comprehensive analysis report can be a comprehensive analysis report of multi-perspective natural language interpretation, which can include but is not limited to technical root cause, business impact, and repair suggestions, and each piece of analysis data can be attached with a confidence score with a confidence range of 0-1 for reference.

[0086] The above technical solution proposes a method for intelligent log analysis and alarm based on a large model. The method realizes log analysis and key event identification steps driven by a large model, continuously updates a key log event list based on key event extraction information, automatically monitors original collected logs in combination with the key log event list, generates real-time log alarm information, and has significant advantages in automation, efficiency, and adaptability of log alarm processing.

[0087] It should be noted that the related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by users or authorized by all parties, and the collection, use, and processing of related data comply with relevant laws, regulations, and standards in relevant regions.

[0088] It should be noted that any arrangement and combination of technical features among the above embodiments also belong to the protection scope of the present application.

[0089] Embodiment Three

[0090] Figure 3 is a schematic diagram of a log alarm device provided by Embodiment Three of the present application, as shown in Figure 3As shown, the device comprises: a log partition module 310, a partition log structured processing module 320, a key log event list generation module 330, and a log alarm information generation module 340, wherein:

[0091] The log partition module 310 is configured to dynamically partition the original collected log according to the log type to obtain a partitioned log.

[0092] The partition log structured processing module 320 is configured to uniformly process the data structure of each type of partitioned log through a large model to obtain a structured partitioned log.

[0093] The key log event list generation module 330 is configured to automatically identify key log events of the structured partitioned log through the large model to obtain a key log event list.

[0094] The log alarm information generation module 340 is configured to monitor the original collected log in real time according to the key log event list to generate real-time log alarm information.

[0095] The embodiment of the application dynamically partitions the original collected log according to the log type to obtain a partitioned log, uniformly processes the data structure of each type of partitioned log through a large model to obtain a structured partitioned log, automatically identifies key log events of the structured partitioned log through the large model to obtain a key log event list, and monitors the original collected log in real time according to the key log event list to generate real-time log alarm information, thereby solving the problems of low efficiency, low automation degree, and low adaptability of the existing log alarm processing procedure, and improving the efficiency, automation degree, and adaptability of log alarm processing.

[0096] Optionally, the log partition module 310 is further configured to identify the log stream source of the original collected log, and dynamically partition the original collected log according to the log stream source of the original collected log to obtain the partitioned log.

[0097] Optionally, the partition log structured processing module 320 is further configured to collect a partitioned log sample from each of the partitioned logs, generate a structured prompt according to the partitioned log sample and a structured data processing template, input the structured prompt into the large model, uniformly process the data structure of the partitioned log sample according to the structured prompt through the large model to obtain a partitioned structured field, extract a field value of the partitioned structured field from the original collected log through a named entity recognition tool, and automatically add the field value to the partitioned structured field to obtain the structured partitioned log.

[0098] Optionally, the partition log structured processing module 320 is further configured to: determine an extended field adapted to the partition log sample according to the type of the partition log sample; generate an extended field example according to the extended field adapted to the partition log sample; and generate the structured prompt according to the partition log sample, the structured data processing template, and the extended field example.

[0099] Optionally, the key log event list generation module 330 is further configured to: generate an initial key event list of the structured partition log; mine log patterns of the structured partition log; generate an event extraction prompt according to the log patterns of the structured partition log, and input the event extraction prompt to the large model to extract a new event list from the log patterns of the structured partition log according to the event extraction prompt through the large model; and update the initial key event list according to the new event list to obtain the key log event list.

[0100] Optionally, the device further comprises a comprehensive analysis report generation module configured to: extract an alarm keyword of the real-time log alarm information, retrieve a target log chain according to the alarm keyword, and generate a multi-dimensional comprehensive analysis report of the real-time log alarm information according to the target log chain.

[0101] The log alarm device can execute the log alarm method provided by any embodiment of the present application, has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the present embodiment can be referred to the log alarm method provided by any embodiment of the present application.

[0102] Since the log alarm device described above is a device that can execute the log alarm method in the embodiments of the present application, based on the log alarm method described in the embodiments of the present application, those skilled in the art can understand the specific implementation of the log alarm device of the present embodiment and its various forms, so the log alarm device how to implement the log alarm method in the embodiments of the present application will not be introduced in detail. As long as the device used to implement the log alarm method in the embodiments of the present application is implemented by those skilled in the art, it belongs to the scope of the present application.

[0103] Embodiment Four

[0104] Figure 4A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0105] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0107] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the log alert method.

[0108] Optionally, the log alarm method can comprise: dynamically partitioning the original collected log according to log types to obtain partitioned logs; uniformly processing data structures of each type of partitioned log through a large model to obtain structured partitioned logs; automatically identifying key log events of the structured partitioned logs through the large model to obtain a key log event list; and monitoring the original collected log in real time according to the key log event list to generate real-time log alarm information.

[0109] In some embodiments, the log alarm method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as storage unit 18. In some embodiments, portions of the computer program, or all of the computer program, can be loaded onto and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. One or more steps of the log alarm method described above can be performed when the computer program is loaded into RAM 13 and executed by processor 11. Alternatively, in other embodiments, processor 11 can be configured, by way of firmware or software modules stored in ROM 12, memory 13, or other storage medium, to perform the log alarm method.

[0110] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0111] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be embodied in whole or in part (including both instructions and data) in a machine, a processor, or on a storage medium.

[0112] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0115] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0116] It should be understood that the various forms of flow shown above can be reordered, steps added or removed. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0117] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log alarm method, characterized in that, include: The original collected logs are dynamically partitioned according to the log type to obtain partitioned logs; By unifying the data structure of each type of partition log through a large model, structured partition logs are obtained. The key log events in the structured partition logs are automatically identified using the large model to obtain a list of key log events; Based on the list of key log events, the original collected logs are monitored in real time, and real-time log alarm information is generated.

2. The method according to claim 1, characterized in that, The process of dynamically partitioning the original collected logs according to the log type to obtain partitioned logs includes: Identify the source of the log stream from the original collected logs; The original collected logs are dynamically partitioned based on the log stream source of the original collected logs to obtain the partitioned logs.

3. The method according to claim 1, characterized in that, The process of unifying the data structure of each type of partition log through a large model to obtain structured partition logs includes: Collect partition log samples from each of the aforementioned partition logs; A structured prompt is generated based on the partition log sample and the structured data processing template, and the structured prompt is input into the large model. The large model then performs unified processing on the data structure of the partition log sample based on the structured prompt to obtain partition structured fields. The field values ​​of the partition structured fields are extracted from the original collection log using a named entity recognition tool and automatically added to the partition structured fields to obtain the structured partition log.

4. The method according to claim 3, characterized in that, The step of generating structured prompts based on the partitioned log sample and structured data processing template includes: Determine the extended fields that the partition log sample should be adapted to based on the type of the partition log sample; Generate extended field examples based on the extended fields adapted from the partition log sample; The structured prompt is generated based on the partition log sample, the structured data processing template, and the extended field example.

5. The method according to claim 1, characterized in that, The process of automatically identifying key log events in the structured partitioned logs using the large model to obtain a list of key log events includes: Generate an initial list of critical events for the structured partitioned log; Mining the log patterns of the structured partitioned logs; An event extraction prompt is generated based on the log pattern of the structured partitioned log, and the event extraction prompt is input into the large model. The large model then extracts a list of new events from the log pattern of the structured partitioned log based on the event extraction prompt. The initial critical event list is updated based on the newly added event list to obtain the critical log event list.

6. The method according to claim 1, characterized in that, After generating the real-time log alarm information, the following is also included: Extract alarm keywords from the real-time log alarm information, and retrieve the target log chain based on the alarm keywords; A multi-dimensional comprehensive analysis report of the real-time log alarm information is generated based on the target log chain.

7. A log alarm device, characterized in that, include: The log partitioning module is used to dynamically partition the raw collected logs according to the log type, resulting in partitioned logs. The partition log structuring module is used to unify the data structure of each type of partition log through a large model to obtain structured partition logs. The critical log event list generation module is used to automatically identify critical log events in the structured partition log through the large model and obtain a critical log event list. The log alarm information generation module is used to monitor the original collected logs in real time based on the list of key log events and generate real-time log alarm information.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the log alarm method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the log alarm method according to any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the log alarm method according to any one of claims 1-6.

Images