Log analysis and early warning method and device based on large language model and medium

Abstract

The invention provides a log analysis and early warning method and device based on a large language model and a medium. The method comprises the following steps: based on a log file monitored by a server in real time, collecting and analyzing the log file according to a preset period of a system, and generating structured information; aggregating the discrete structured information with the space-time correlation information to generate an information panorama; designing cue words to execute image analysis and image reasoning; and importing the information panorama into a predetermined language model, and generating and outputting an interpretable early warning report by using the cue word. The invention discloses a log analysis and early warning device based on a large language model, electronic equipment and a computer readable storage medium.

Description

Technical Field

[0001] This invention relates to the field of log analysis technology, and in particular to a log analysis and early warning method, device and medium based on a large language model. Background Technology

[0002] In modern enterprise-level Java application operations and maintenance, Oracle Weblogic server serves as a critical application middleware, and its runtime stability is of paramount importance. Garbage Collection (GC) logs are the most direct and comprehensive built-in data source for diagnosing JVM heap memory health and predicting the risk of memory overflow. They record in detail the behavior and duration of each GC event, as well as the capacity changes of various heap memory regions (young generation, old generation, etc.) before and after collection. However, GC log data is multi-dimensional, complex, and continuously generated.

[0003] In existing technologies, the following problems often exist: (1) Rule-based and static threshold-based indicator monitoring method: This is the most common implementation method at present. Operation and maintenance personnel parse several core indicators from GC logs and set independent static alarm thresholds for each indicator. The problem with this method is that: First, GC behavior is a complex process in which multiple indicators are related and work together. It is easy to misjudge if a single indicator is viewed in isolation. For example, if the usage rate of the young generation is high but accompanied by stable object reclamation, it is not necessarily a risk; on the contrary, if the old generation GC is frequent but the efficiency of a single reclamation is low, it may indicate a more hidden memory leak. Separating the indicators makes it impossible to capture this multi-dimensional correlation. Second, static thresholds cannot adapt to the complex and ever-changing load patterns of applications, and are prone to false alarms during peak business periods or false alarms in slow memory leak scenarios. (2) Time-series prediction method based on traditional machine learning models: This is a more advanced technology. This method extracts a series of features from historical GC logs (such as the mean, variance, and growth trend of memory region usage, GC frequency, etc.) to construct a time-series dataset and trains a prediction model (such as LSTM or Prophet) to predict whether a key indicator (usually heap memory usage) will exceed a safe threshold in the future. The limitations of this method are: First, it heavily relies on a large amount of high-quality, labeled (i.e., data on whether memory overflow has occurred) historical data for model training, which is often difficult to obtain in real production environments. Second, the model's input is usually structured, numerical feature vectors, requiring extensive and professional feature engineering, and the quality of feature selection directly affects the model's performance. Third, the model is a "black box," and its predictions (such as "80% probability of overflow in the next 2 hours") lack interpretability; operations personnel cannot understand the reasoning behind them, making it difficult to quickly locate the root cause and take targeted measures. This disclosure aims to find an innovative technical solution to address the above problems. Summary of the Invention

[0004] This invention provides a log analysis and early warning method, device and medium based on a large language model, which can generate risk warnings by using a panoramic view combined with large model analysis, greatly reducing human error in transmission.

[0005] Firstly, a log analysis and early warning method based on a large language model is provided, comprising: collecting and parsing log files based on real-time monitoring of server log files according to a predetermined system cycle to generate structured information; aggregating the discrete structured information with spatiotemporal correlation information to generate an information panorama; designing prompt words to perform image analysis and image reasoning; importing the information panorama into a predetermined language model, and using the prompt words to generate and output an interpretable early warning report.

[0006] In some embodiments, based on log files monitored in real time by the server, the log files are collected and parsed according to a predetermined system period, including the following steps: obtaining the raw data of the log files from the server according to a predetermined system period; parsing the raw data of the log files based on predefined regular expressions to generate the structured information, wherein the structured information includes at least a timestamp field and a data content field.

[0007] In some embodiments, the discrete structured information with spatiotemporal correlation information is aggregated to generate an information panorama, including the following steps: based on the structured information, statistical analysis of the same spatiotemporal correlation information within a specific time period is performed using a predefined script analysis language of the system; based on time changes, specific indicators are selected to generate the information panorama; and the relevant structured information used to generate the information panorama is encapsulated to form a text replacement of the information panorama.

[0008] In some embodiments, designing prompts to perform image analysis and image reasoning includes the following steps: designing prompts to describe the patterns of the information panorama and analyzing the interactive relationships of the structured information; designing prompts to describe the risks of the information panorama and analyzing the risk expectations of the structured information; wherein, the risk expectations include risk identification and risk assessment.

[0009] In some embodiments, importing the information panorama into a predetermined language model and using the prompt words to generate and output an interpretable warning report includes the following steps: importing the prompt words and the panorama using the predetermined language model; collecting information from the pattern field, risk prompt field, and risk cause field output by the language model; and generating the warning report.

[0010] In some embodiments, the method further includes: pushing the warning report out through multiple channels, wherein the channels include at least email and SMS.

[0011] Secondly, a log analysis and early warning device based on a large language model is provided, comprising: an information collection module, used to collect and parse log files based on real-time monitoring of server log files according to a predetermined system cycle to generate structured information; a panoramic image generation module, used to aggregate the discrete structured information with spatiotemporal correlation information to generate an information panoramic image; a prompt word module, used to design prompt words to perform image analysis and image reasoning; and a report output module, used to import the information panoramic image into a predetermined language model, and use the prompt words to generate and output an interpretable early warning report.

[0012] In some embodiments, the system further includes an information push module for pushing the warning report out through multiple channels, wherein the channels include at least email and SMS.

[0013] Thirdly, an electronic device is provided, characterized in that it includes: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to utilize the at least one processor to cause the electronic device to execute the above-described log analysis and early warning method based on a large language model.

[0014] Fourthly, a computer-readable storage medium is also provided, characterized in that the computer-readable storage medium stores program code, which can be called and executed by a processor: the log analysis and early warning method based on a large language model as described in any one of claims 1 to 6.

[0015] Compared with existing technologies, the present invention can achieve at least the following beneficial effects: it can generate risk warnings by using panoramic images in combination with large model analysis, which greatly reduces human error in transmission.

[0016] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify essential or necessary features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0018] Figure 1This illustration shows a schematic diagram of log analysis and early warning based on a large language model provided in an embodiment of this application; Figure 2 A schematic diagram of a log analysis and early warning device based on a large language model provided in an embodiment of this application is shown; Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0020] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] This application provides a log analysis and early warning method based on a large language model. Please refer to [link / reference]. Figure 1 This figure is a schematic diagram of the first embodiment of this application. The following is in conjunction with... Figure 1 The first embodiment of this application provides a detailed description of a log analysis and early warning method 100 based on a large language model.

[0022] Step S102: Obtain log information, that is, based on the log files monitored in real time on the server, collect and parse the log files according to the system's predetermined cycle to generate structured information. For example, the GC log collection and parsing module obtains the raw GC log files from the target Weblogic application server in real time. The logs are parsed to extract the key indicators of each GC event, forming a structured time-series data stream. The GC log files (such as gc.log) are monitored in real time by an Agent deployed on the Weblogic server side, and incremental collection is reported to the LogEasy platform in real time. Parsing: The LogEasy parser matches each line of log based on a predefined regular expression pattern. For example, for the G1 GC log, matching \[Eden.*->.*B.*\] is used to parse young generation reclamation information; matching \[Full GC.*\] is used to identify Full GC events, and key values ​​such as used_before, used_after, and duration are extracted from them. The parsing results are converted into a standardized JSON object, containing timestamps and all extracted indicator fields.

[0023] Step S104: Generate an information panorama, that is, aggregate the discrete structured information with spatiotemporal correlation to generate an information panorama. For example, the process of generating an information panorama may include: window aggregation, indicator selection, chart generation, and data encapsulation. Specifically, window aggregation: using the LogEasy SPL analysis language, a time window (such as the most recent day) is statistically analyzed, and each indicator is bucketed (e.g., 5-minute buckets); indicator selection: select several core indicators with the most diagnostic value, such as memory size before and after old generation garbage collection, memory size before and after young generation garbage collection, number of GC collections, GC collection time, etc.; chart generation: use LogEasy's multi-Y-axis trend chart. Create an independent Y-axis for each selected indicator, and draw its time-varying line using different colors and line styles. To enhance readability, legends, axis labels, and grids need to be added. Finally, render the chart as a high-resolution bitmap (such as PNG format); data encapsulation: encapsulate the image generated by the trend chart and provide detailed structured data description text as "text replacement for the chart".

[0024] Step S106: Design prompts, that is, design prompts to perform image analysis and image reasoning. For example, a prompt is a carefully designed text template that is dynamically populated with the current chart data during each analysis. An example template is shown below: You are a JVM performance tuning and troubleshooting expert with 10 years of experience. Your task is to analyze a timing chart that reflects the recent memory health of a Java application.

[0025] [Chart Description] This is a multi-curve time series chart showing the changes in the following four key GC metrics every 5 minutes over the past day: 1. Old generation memory usage (unit: percentage, red curve).

[0026] 2. Recycling frequency of the younger generation (unit: times / minute, blue curve).

[0027] 3. Time taken for a single Full GC (unit: milliseconds, green curve).

[0028] 4. Total heap memory usage (unit: megabytes, yellow curve).

[0029] The X-axis represents time. The specific numerical details of the chart are as follows: [Insert image and detailed structured data point description].

[0030] Please perform the following analysis and output JSON. 1. **Pattern Description:** Summarize the overall trend of each curve in the graph and the interaction between them in one sentence.

[0031] 2. **Risk Identification:** Based on your expertise, determine if the graph contains any abnormal patterns or combinations of risk indicators suggesting an impending OutOfMemoryError. Please specify which patterns you are referring to.

[0032] 3. **Risk Assessment:** Predict the probability of a memory overflow occurring within the next 5-10 minutes. Use "High," "Medium," and "Low" to represent the risk level, and provide a confidence score between 0 and 100.

[0033] 4. **Root Cause Inference**: If risks exist, infer the most likely root cause based on observed patterns (e.g., memory leaks, large object allocations, inappropriate GC strategies, etc.).

[0034] Please strictly adhere to the following JSON format when outputting your analysis results, and do not include any other explanatory text: { "pattern_description": "string", "risk_indicators": ["string1", "string2", ...], "risk_level": "high / medium / low", "confidence": integer, "root_cause_hypothesis": "string", "suggested_actions": ["string1", "string2", ...] } Step S108: Generate an alarm. This involves importing the information panorama into a predetermined language model and using the prompt words to generate and output an interpretable early warning report. For example, the structured output of the model is converted into actionable operational instructions. Result parsing and judgment: The risk_level and confidence fields in the JSON are parsed. When the risk level is "high" and the confidence level is greater than a preset threshold (e.g., 75%), the system determines that an immediate warning needs to be issued. Report synthesis: All fields in the JSON are reorganized using fluent natural language to form a complete report. For example, pattern_description, risk_indicators, and root_cause_hypothesis are merged into a coherent "Analysis Basis" paragraph.

[0035] In some embodiments, based on log files monitored in real time by the server, the log files are collected and parsed according to a predetermined system period, including the following steps: obtaining the raw data of the log files from the server according to a predetermined system period; parsing the raw data of the log files based on predefined regular expressions to generate the structured information, wherein the structured information includes at least a timestamp field and a data content field.

[0036] In some embodiments, the discrete structured information with spatiotemporal correlation information is aggregated to generate an information panorama, including the following steps: based on the structured information, statistical analysis of the same spatiotemporal correlation information within a specific time period is performed using a predefined script analysis language of the system; based on time changes, specific indicators are selected to generate the information panorama; and the relevant structured information used to generate the information panorama is encapsulated to form a text replacement of the information panorama.

[0037] In some embodiments, designing prompts to perform image analysis and image reasoning includes the following steps: designing prompts to describe the patterns of the information panorama and analyzing the interactive relationships of the structured information; designing prompts to describe the risks of the information panorama and analyzing the risk expectations of the structured information; wherein, the risk expectations include risk identification and risk assessment.

[0038] In some embodiments, importing the information panorama into a predetermined language model and using the prompt words to generate and output an interpretable warning report includes the following steps: importing the prompt words and the panorama using the predetermined language model; collecting information from the pattern field, risk prompt field, and risk cause field output by the language model; and generating the warning report.

[0039] In some embodiments, the method further includes: pushing the warning report out through multiple channels, wherein the channels include at least email and SMS. For example, in an embodiment, multiple channels can be used for push: the system has a built-in notification adapter that can push the final report to the corresponding responsible person via Webhook, SMS, syslog, HTTP, Email, etc.

[0040] This application also provides a log analysis and early warning device 200 based on a large language model. Please see below. Figure 2 This figure is a schematic diagram of the second embodiment of this application. The following is in conjunction with... Figure 2 This application provides a detailed description of a log analysis and early warning device 200 based on a large language model, according to the first embodiment of the present application. The device 200 may include: Information collection module 202: Used to collect and parse the log files based on the real-time monitoring log files of the server according to the system's predetermined cycle, and generate structured information.

[0041] Panoramic image generation module 204: used to aggregate the discrete structured information with spatiotemporal correlation information to generate an information panoramic image.

[0042] Prompt word module 206: Used to design prompt words to perform image analysis and image reasoning.

[0043] Report output module 208: Used to import the information panorama into a predetermined language model, and generate and output an interpretable warning report using the prompt words.

[0044] In some embodiments, the system further includes an information push module for pushing the warning report out through multiple channels, wherein the channels include at least email and SMS.

[0045] like Figure 3 As shown, an electronic device provided in this embodiment of the invention may include a processor 320 and a memory 310. Optionally, the electronic device may further include a transceiver. The processor, memory, and transceiver may be connected via a communication bus. The memory stores computer-readable instructions, which, when executed by the processor, implement the steps described above for the log system client method. Similarly, an electronic device provided in this embodiment of the invention may include a processor and a memory. Optionally, the electronic device may further include a transceiver. The processor, memory, and transceiver may be connected via a communication bus. The memory stores computer-readable instructions, which, when executed by the processor, implement the steps described above for the log analysis and alerting method based on a large language model. Further details are omitted here.

[0046] In a specific implementation, as one example, processor 320 may include one or more CPUs.

[0047] Optionally, in a specific implementation, if the memory 310, processor 320 and communication interface 330 are integrated on a single chip, the memory 310, processor 320 and communication interface 330 can communicate with each other through an internal interface.

[0048] In a specific implementation, as one example, the electronic device may also include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0049] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can be referred to the above method embodiment, which will not be repeated here.

[0050] A transceiver is used to communicate with network devices or with terminal devices.

[0051] Optionally, the transceiver may include a receiver and a transmitter. The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0052] Optionally, the transceiver can be integrated with the processor or exist independently and coupled to the processor through the interface circuit of the electronic device. This embodiment of the invention does not specifically limit this.

[0053] It should be noted that the structure of the electronic device described above does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Furthermore, the technical effects of the electronic device can be referred to the technical effects of the above method embodiments, and will not be repeated here.

[0054] In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement steps such as those in a log analysis and alerting method based on a large language model. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0055] This invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, which can be called by a processor to execute the above-described log analysis and early warning method based on a large language model.

[0056] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0057] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0058] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0062] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A log analysis and early warning method based on a large language model, comprising: based on the server real-time monitoring log file, according to the system predetermined period collection and parsing of the log file, generate structured information; The discrete structured information with space-time correlation information is aggregated to generate an information panorama; Design prompt words to perform image analysis and image reasoning; The information panorama is imported into a predetermined language model, and an interpretable early warning report is generated and output by using the prompt words.

2. The log analysis and early warning method based on a large language model according to claim 1, characterized in that, Based on the server real-time monitoring log file, according to the system predetermined period collection and parsing of the log file, comprising the following steps: According to the system predetermined period from the server to obtain the original data of the log file; Based on the predefined regular expression, parse the original data of the log file to generate the structured information, wherein the structured information at least contains timestamp field and data content field. 3.The log analysis and early warning method based on large language model according to claim 1, characterized in that, The discrete structured information with space-time correlation information is aggregated to generate an information panorama, comprising the following steps: Based on the structured information, using the system predetermined script analysis language, the same space-time correlation information in a specific time period is counted; Based on the time change, select specific indicators to generate the information panorama; The related structured information of the information panorama is data encapsulated to form the text replacement of the information panorama. 4.The log analysis and early warning method based on a large language model according to claim 1, characterized in that, Design prompt words to perform image analysis and image reasoning, comprising the following steps: Design prompt words to describe the mode of the information panorama, analyze the interaction relationship of the structured information; Design prompt words to describe the risk of the information panorama, analyze the risk expectation of the structured information; wherein the risk expectation includes risk identification and risk assessment. 5.The log analysis and early warning method based on large language model according to claim 1, characterized in that, The information panorama is imported into a predetermined language model, and an interpretable early warning report is generated and output by using the prompt words, comprising the following steps: Using the predetermined language model, import the prompt words and the panorama; Collect the information of the mode field, risk prompt field and risk reason field output by the language model to generate the early warning report.

6. The log analysis and early warning method based on a large language model according to any one of claims 1-5, characterized in that, Further comprising: The early warning report is pushed out through multiple channels, wherein the channels at least include email, short message.

7. A log analysis and early warning device based on a large language model, comprising: An information collection module for collecting and parsing log files based on server real-time monitoring according to the system predetermined period, generating structured information; Panorama generation module, for aggregating discrete structured information with space-time correlation information to generate an information panorama; Prompt word module, for designing prompt words to perform image analysis and image reasoning; The report output module is used for importing the information panorama into a predetermined language model, and an interpretable early warning report is generated and output by using the prompt words.

8. The log analysis and early warning device based on a large language model according to claim 7, characterized in that, Further comprising: The information push module is used for pushing the early warning report out through multiple channels, wherein the channels at least include email, short message.

9. An electronic device, comprising: Comprising: At least one processor; And At least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the electronic device to perform: the log analysis and early warning method based on a large language model according to any one of claims 1-6.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, and the program code can be called and executed by the processor: the log analysis and early warning method based on a large language model according to any one of claims 1-6.

Images