A log analysis method, device, apparatus and storage medium

Abstract

The application discloses a log analysis method and device, equipment and a storage medium, and relates to the technical field of data analysis, and comprises the following steps: converting collected multi-source heterogeneous logs into structured data in a preset format, and storing the structured data into a search engine; receiving input information of a user through the search engine, analyzing the input information to obtain an analyzed keyword; matching the analyzed keyword with the structured data stored in the search engine to obtain matched data; determining a log analysis result based on the matched data, a local large model and a preset log analysis instruction, and displaying the log analysis result. Therefore, the application can upgrade traditional keyword retrieval to intelligent question answering and step-by-step guidance, reduce the analysis threshold, and greatly improve the fault positioning efficiency.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a log analysis method, apparatus, device, and storage medium. Background Technology

[0002] As enterprise IT (Internet Technology) infrastructure continues to expand, the log data generated by servers is characterized by massive volume, heterogeneity, and rapid growth, which greatly hinders log retrieval.

[0003] The industry faces many problems in log retrieval: 1. The search efficiency is low. Test and maintenance personnel need to log in to multiple servers via SSH (Secure Shell) and manually search using command line tools such as grep and awk. The process is cumbersome and time-consuming, and it is not possible to perform cross-host or cross-log type related queries. 2. The analysis threshold is high, and the log formats vary greatly (system logs, application logs, hardware BMC (Baseboard Management Controller) logs, etc.). Understanding their meaning requires deep professional knowledge and experience, and the training period for newcomers is long. 3. Fault localization is difficult. When a fault occurs, it is often accompanied by a large number of related and unrelated log entries. Manually sorting out the root cause clues (fault chain) from the noise is like finding a needle in a haystack. The average repair time is long and the risk of business interruption is high. 4. Existing tools are difficult to deploy. Some mature log management platforms (such as ELK (Elasticsearch + Logstash + Kibana, distributed storage and full-text search in the background) / EFK (Elasticsearch + Filebeat + Kibana, log collection system) complete stack) are powerful, but their deployment is complex and resource consumption is high, which is not friendly to small and medium-sized clusters or edge computing scenarios.

[0004] Therefore, developing a lightweight, easy-to-deploy, and intelligent log analysis system that can uniformly process various logs, lower the analysis threshold, and assist in rapid fault location is of great practical significance and commercial value for improving the level of automation and reliability assurance capabilities of testing. Summary of the Invention

[0005] The purpose of this invention is to provide a log analysis method, apparatus, device, and storage medium that upgrades traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency. The specific solution is as follows: In a first aspect, the present invention discloses a log analysis method, comprising: The collected multi-source heterogeneous logs are converted into structured data in a preset format, and the structured data is stored in the search engine. The system receives user input information through a search engine, parses the input information, and obtains the parsed keywords. The parsed keywords are matched with the structured data stored in the search engine to obtain the matched data; The log analysis results are determined based on the matched data, the local large model, and the preset log analysis commands, and then displayed.

[0006] Optionally, before converting the collected multi-source heterogeneous logs into structured data in a preset format, the following steps are also included: Deploy lightweight collectors on each server; Develop parsing plugins for each log type; the parsing plugins include preset regular expressions or delimiter rules; Collect multi-source heterogeneous logs using a lightweight collector; Accordingly, the collected multi-source heterogeneous logs are converted into structured data in a preset format, including: The parsing plugin is invoked to parse multi-source heterogeneous logs and convert the corresponding parsing results into structured data in a preset format; The default format includes a timestamp, the IP address of the host that generated the log, the log source, the original log text, and the event type.

[0007] Optionally, before storing structured data in the search engine, the following may also be included: The target deployment method is determined based on the amount of structured data; the target deployment methods include single-node deployment and minimum-scale cluster deployment. The search engine is deployed to the server based on the target deployment method.

[0008] Optionally, the parsed keywords include time, object, and event; Accordingly, the parsed keywords are matched with the structured data stored in the search engine to obtain the matched data, including: Based on the parsed keywords, time-based and fuzzy searches are performed on the structured data stored in the search engine to obtain the corresponding matched data.

[0009] Optionally, the parsed keywords are matched with the structured data stored in the search engine. After obtaining the matched data, the following steps are also included: The matched data is associated according to the timeline and preset rules to obtain associated data, which is then sent to the local large model through the search engine.

[0010] Optionally, the log analysis results are determined based on the matched data, the local large model, and preset log analysis commands, including: The search engine sends the correlated data and relevant expertise to the local large model, so that the local large model can output the corresponding log analysis results based on the correlated data and preset log analysis instructions.

[0011] Optionally, the log analysis results can be displayed, including: An answer summary is generated based on the log analysis results, and the answer summary is displayed. The log entries associated with the log analysis results are highlighted, and the order of occurrence of the events associated with the log analysis results is displayed in a chart format.

[0012] Secondly, the present invention discloses a log analysis device, comprising: The data storage module is used to convert the collected multi-source heterogeneous logs into structured data in a preset format and store the structured data in the search engine. The parsing module is used to receive user input information through the search engine, parse the input information, and obtain the parsed keywords; The data matching module is used to match the parsed keywords with the structured data stored in the search engine to obtain the matched data; The results display module is used to determine the log analysis results based on the matched data, the local large model, and the preset log analysis instructions, and to display the log analysis results.

[0013] Thirdly, the present invention discloses an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute computer programs to perform the steps of the log analysis method described above.

[0014] Fourthly, the present invention discloses a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the aforementioned log analysis method.

[0015] In performing log analysis, this invention first converts the collected multi-source heterogeneous logs into structured data in a preset format and stores the structured data in a search engine. Next, it receives user input information through the search engine, parses the input information to obtain parsed keywords, and then matches the parsed keywords with the structured data stored in the search engine to obtain matched data. Finally, based on the matched data, the local large model, and preset log analysis instructions, the log analysis results are determined and displayed.

[0016] Beneficial Effects: This invention converts multi-source, heterogeneous logs into standardized, structured data in real time. Standardized log data is stored in a search engine, and all subsequent analyses are based on unified semantic fields, masking underlying format differences. The engine can receive user-generated questions or fault descriptions in natural language and collaboratively execute them using a rule engine and a locally deployed lightweight local large-scale model. This eliminates the need for testers to memorize complex command-line syntax or query numerical user lines; they can perform efficient retrieval and preliminary analysis using natural language. An interactive guided interface integrates expert experience into the process, assisting novices in quickly locating problems. It breaks down information silos across logs and hosts, reducing manual correlation work that previously took hours to minutes. Combining the determinism of the rule engine with the semantic understanding and reasoning capabilities of the large language model, it can discover potential, indirect relationships, reducing false positives and false negatives. It upgrades traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency. Attached Figure Description

[0017] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a log analysis method provided in an embodiment of the present invention; Figure 2 A schematic diagram of a log analysis device provided in an embodiment of the present invention; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0020] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0021] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In some technologies, log analysis platforms based on the traditional ELK / EFK stack employ Elasticsearch for log storage and retrieval, Logstash or Fluentd for log collection and parsing, and Kibana for visualization. While these platforms are feature-rich and have active communities, they are resource-intensive to deploy, complex to configure, and lack interactivity. Graylog, an open-source log management platform, offers a more integrated log processing workflow compared to the ELK stack, but still has room for improvement in intelligent analysis and ease of use. Loki (by Grafana Labs), inspired by Prometheus, focuses on log indexing and querying, with a lighter design, but its ecosystem and advanced analytics capabilities are still under development. These applications either emphasize storage and retrieval or fixed-rule alerting, and have not yet formed a mature and efficient solution combining lightweight deployment, unified semantic understanding, and intelligent interactive analysis. To address these technical issues, this invention discloses a log analysis method, apparatus, device, and storage medium that upgrades traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency.

[0023] See Figure 1 As shown, this embodiment of the invention provides a log analysis method, including: Step S11: Convert the collected multi-source heterogeneous logs into structured data in a preset format, and store the structured data in the search engine.

[0024] In this embodiment of the invention, a lightweight collector (such as Fluent Bit) and a customizable parsing plugin are first used to convert multi-source heterogeneous logs into structured data conforming to a unified log model in real time. Therefore, before converting the collected multi-source heterogeneous logs into structured data in a preset format, a lightweight collector is deployed on each server; parsing plugins corresponding to each log type are developed; the parsing plugins include preset regular expressions or delimiter rules; multi-source heterogeneous logs are collected using the lightweight collector; correspondingly, the collected multi-source heterogeneous logs are converted into structured data in a preset format, including: calling the parsing plugin to parse the multi-source heterogeneous logs and converting the corresponding parsing results into structured data in a preset format; wherein, the preset format includes a timestamp, the IP address of the host that generated the log, the log source, the original log text, and the event type. The parsing plugin is a pluggable software module containing parsing rules specific to a certain log format (such as Syslog, JSON, Nginx logs), used to extract structured fields from the original log text.

[0025] Specifically, a lightweight Fluent Bit collector is deployed on the server. A corresponding parsing plugin is developed for each log type to be collected (e.g., message and dmesg, sel logs, smart logs, RAID (Redundant Arrays of Independent Disks) card logs). Each plugin contains specific regular expressions or delimiter rules. The data output by the parsing plugin must conform to a predefined "unified log model." Furthermore, the output data must include necessary fields for log backtracking, differentiation, and retrieval: the log event occurrence time (timestamp), the host IP address that generated the log (host.ip), the log source (message, dmesg, smartctl, RAID, etc.), the event type (diskerror, etc.), the severity level (INFO, WARN, ERROR, CRITCAL, etc.), the original log text (message.log, etc.), and the parsed structured details (user password, etc.). The unified log model is an intermediate data structure defined in this invention, used to map raw logs from different sources and in different formats to a standard format with unified semantic fields. In other words, regardless of the original appearance of the logs, the output will be uniformly formatted as JSON and must include some key fields.

[0026] After converting the collected multi-source heterogeneous logs into structured data in a preset format, the structured data needs to be stored in the search engine. Before storing the structured data in the search engine, the target deployment method is determined based on the volume of the structured data. Target deployment methods include single-node deployment and minimum-scale cluster deployment. The search engine is then deployed to the server based on the target deployment method. The standardized JSON (JavaScript Object Notation) format data is then passed to the Elasticsearch cluster. Depending on the data volume, a single-node or minimum-scale cluster deployment can be used. The processed JSON data from the previous step is then sent to the Elasticsearch database. Elasticsearch will index this data, making subsequent queries (especially those based on keywords and time ranges) very fast. Depending on the log volume, a cluster can be formed using only one or a few nodes. Single-node deployment: Only one Elasticsearch instance is installed on one server. Suitable scenarios: Small data volume (e.g., a few GB per day); small company size; few servers (a few to dozens); test environment or initial trial; low high availability requirements. Minimal cluster deployment: This involves using multiple servers (typically 3) to form a cluster. Suitable scenarios: Large data volumes (tens to hundreds of GB per day); production environments with numerous servers (hundreds of servers); high availability requirements (service continues even if one or two machines fail); and better query performance. This approach allows for cost control: using a cluster would be a waste of resources for small data volumes; smooth upgrades: you can start with a single-node deployment and expand to a cluster later as the data volume increases; and improved adaptability: this solution can be used by enterprises of different sizes. In short, Elasticsearch deployment is not fixed; it depends on your actual data volume. Use a simple single-machine setup for small amounts of data, and a cluster for large amounts of data—choose flexibly.

[0027] Step S12: Receive user input information through the search engine, parse the input information to obtain the parsed keywords.

[0028] In this embodiment of the invention, a search engine receives user input information and then parses it. The parsed keywords include time, object, and event. In a specific embodiment, if a user asks "Did the server experience any hard drive-related errors yesterday?", the engine first receives and parses the received natural language. Taking the user's question as an example, it extracts the keywords "yesterday," "hard drive," and "error."

[0029] Step S13: Match the parsed keywords with the structured data stored in the search engine to obtain the matched data.

[0030] In this embodiment of the invention, time-based and fuzzy searches are performed on the structured data stored in the search engine based on the parsed keywords to obtain the corresponding matched data. Then, the matched data is correlated according to the timeline and preset rules to obtain correlated data, which is then sent to the local large model via the search engine. Specifically, time (yesterday's date) and fuzzy searches are performed on the logs based on the extracted keywords (checking for "disk" in the event type and "fail," "error," and "warning" in the structured details). The engine then retrieves the relevant logs and performs preliminary correlation according to preset rules: for example, aligning "disk I / O error" warnings and "sdb failed" errors occurring within the same time period on the timeline. In other words, logs are searched in Elasticsearch based on the extracted keywords. For example, "yesterday" is converted to a specific date, and "disk" is searched in the event_type field, while "fail," "error," etc., are fuzzily searched in the structured_details field. Then, the scattered logs are concatenated according to the timeline and rules. For example, the log entries "disk I / O error" and "sdb failed" that occurred at adjacent times can be associated as a single complete event.

[0031] Step S14: Determine the log analysis results based on the matched data, the local large model, and the preset log analysis instructions, and display the log analysis results.

[0032] In this embodiment of the invention, when determining the log analysis result based on the matched data, the local large model, and preset log analysis instructions, the search engine sends the associated data and corresponding professional knowledge to the local large model, so that the local large model can output the corresponding log analysis result according to the associated data and preset log analysis instructions. That is, the engine submits the associated key log entries (including context) and professional knowledge (e.g., hot-swapping a hard drive on a backplane that does not support hot-swapping will cause a hard drive error) to the locally deployed lightweight LLM (Large Language Model), and simultaneously provides the instruction: Please analyze the cause of the hard drive error based on the following server log sequence, and provide the most likely root cause ranking and brief evidence. In summary, the large language model sends the associated logs and related professional knowledge (e.g., "hot-swapping a hard drive on a backplane that does not support hot-swapping will cause an error") to a local large model. The instruction is: Please analyze the root cause and rank it.

[0033] Finally, an answer summary is generated based on the log analysis results and displayed. Log entries associated with the log analysis results are highlighted, and the order of events associated with the log analysis results is shown in a chart. In this invention, on the web interface, the system provides a simple text answer, including: an answer summary (e.g., displaying: "Yesterday there was a hard drive error, the root cause may be hot-swapping the hard drive, evidence as follows..."), a list of key evidence (highlighting log entries associated with the hard drive error), a visual timeline (showing the order of events in a chart), and one-click collection (collecting the original text of the error log with one click). In a specific embodiment, on the web interface, the user will see: Answer Summary: Summarize the result in one sentence. For example: "Yesterday, the server did experience a hard drive error, most likely caused by a hot-swapping operation, evidence as follows..."

[0034] Key evidence list: Highlights specific log entries related to the conclusion.

[0035] Visualized timeline: Use charts to visually display the time sequence of the entire failure (e.g., warning first, then error, and finally recovery).

[0036] One-click collection: Provides a button to download the relevant raw logs in one click, making it convenient for archiving or further manual analysis.

[0037] Furthermore, this invention can be further optimized. Firstly, it can be extended to full-stack observability, expanding the unified log model into a "unified observability model" that incorporates metrics and traces data. The system can not only analyze logs but also automatically correlate performance indicator curves and distributed call traces for the corresponding time period when anomalies are detected, achieving true root cause localization. That is, it no longer only looks at log data but also incorporates metrics and traces, providing a comprehensive view of the system's health. Secondly, it integrates the analysis chain with IT service management processes. When a user submits a fault ticket, the system can automatically trigger the log analysis chain, pre-filling possible causes and discovered evidence, greatly improving the processing speed for frontline operations personnel. Thus, automatic triggering occurs when a user submits a fault ticket (e.g., "Server A is slow"). Background pre-analysis occurs when the ticket system automatically calls the log analysis engine to quickly analyze the logs, metrics, and traces of "Server A" around the "current time point." Intelligent population: In addition to the user-entered information, the work order system automatically adds a section called "System Pre-analysis Results," which is pre-filled with: possible causes (e.g., root cause probability ranking: 1. High CPU (Central Processing Unit) load; 2. Database connection pool full). Found evidence (e.g., related logs, etc.) and abnormal metrics (e.g., CPU utilization 99%). This significantly shortens the fault diagnosis time.

[0038] Beneficial Effects: This invention converts multi-source, heterogeneous logs into standardized, structured data in real time. Standardized log data is stored in a search engine, and all subsequent analyses are based on unified semantic fields, masking underlying format differences. The engine can receive user-generated questions or fault descriptions in natural language and collaboratively execute them using a rule engine and a locally deployed lightweight local large-scale model. This eliminates the need for testers to memorize complex command-line syntax or query numerical user lines; they can perform efficient retrieval and preliminary analysis using natural language. An interactive guided interface integrates expert experience into the process, assisting novices in quickly locating problems. It breaks down information silos across logs and hosts, reducing manual correlation work that previously took hours to minutes. Combining the determinism of the rule engine with the semantic understanding and reasoning capabilities of the large language model, it can discover potential, indirect relationships, reducing false positives and false negatives. It upgrades traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency.

[0039] In summary, this invention provides a lightweight log analysis and fault location assistance system and method that supports multiple log types and possesses interactive intelligent analysis capabilities. The core content lies in: standardizing heterogeneous logs by defining a unified log model as a "translation hub"; and then designing an interactive analysis chain driven by a rule engine and a lightweight LLM, upgrading traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency.

[0040] The specific architecture of this invention includes the following parts: 1. Unified Log Collection and Parsing Layer: This layer uses a lightweight collector (such as Fluent Bit) and customizable parsing plugins to convert multi-source, heterogeneous logs into structured data conforming to a unified log model in real time. Specifically, it deploys Fluent Bit's lightweight collector on the server and develops a corresponding parsing plugin for each log type that needs to be collected. Each plugin includes specific regular expressions or delimiter rules. The data output by the parsing plugin must adhere to a predefined "unified log model." Furthermore, the output data must include necessary fields for log backtracking, differentiation, and retrieval: the log event's occurrence time, the host IP address that generated the log, the log source, the event type, the severity level, the original log text, and the parsed structured details.

[0041] 2. Storage and Indexing: Standardized log data is stored in the search engine, and all subsequent analysis is based on unified semantic fields, shielding the underlying format differences. The standardized JSON format data is then fed into the Elasticsearch cluster, which can be deployed using a single node or a minimal cluster depending on the data volume.

[0042] 3. Interactive Intelligent Analysis Engine: This engine receives user questions or fault symptoms expressed in natural language, breaks them down into a series of retrieval, association, and analysis steps (analysis chain), and calls upon the rule engine and a locally deployed lightweight LLM to execute them collaboratively. The engine receives and parses the received natural language, extracts keywords from the user's question, and performs time-based and fuzzy searches in the logs based on the extracted keywords. Then, the engine obtains the relevant logs found in the query and performs preliminary associations according to preset rules. Finally, the engine submits the associated key log entries (including context) and professional knowledge to the locally deployed lightweight LLM, while issuing the instruction: Please analyze the cause of the hard drive error based on the following server log sequence, and provide the most likely root cause ranking and brief evidence.

[0043] 4. Guided Fault Location Interface: Provides a concise web-based interactive interface. The system not only returns answers but also proactively reveals the analysis process, provides a chain of evidence, and offers suggestions for further investigation through a multi-turn dialogue. In the web interface, the system provides a simple text response, including: an answer summary (e.g., "Yesterday there was a hard drive error, the root cause may be hot-swapping the hard drive, evidence as follows..."), a list of key evidence (highlighting log entries related to the hard drive error), a visual timeline (showing the sequence of related events in chart form), and one-click collection (collecting the original text of the error log with one click).

[0044] This significantly lowers the barrier to entry for users: testers no longer need to memorize complex command-line syntax or query DSLs (Domain Specific Languages); they can perform efficient searches and preliminary analyses using natural language. The interactive guided interface integrates expert experience into the process, helping novices quickly locate problems. It also improves the efficiency and accuracy of log inspection: through a unified model and intelligent association, it breaks down information silos across logs and hosts, reducing manual association work that previously took hours to minutes. Combined with the deterministic nature of the rule engine and the semantic understanding and reasoning capabilities of the LLM, it can discover potential, indirect relationships, reducing false positives and false negatives.

[0045] See Figure 2 As shown, an embodiment of the present invention provides a log analysis device, including: Data storage module 11 is used to convert the collected multi-source heterogeneous logs into structured data in a preset format and store the structured data in the search engine; The parsing module 12 is used to receive user input information through the search engine, parse the input information, and obtain the parsed keywords; Data matching module 13 is used to match the parsed keywords with the structured data stored in the search engine to obtain the matched data; The results display module 14 is used to determine the log analysis results based on the matched data, the local large model, and the preset log analysis instructions, and to display the log analysis results.

[0046] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0047] Beneficial Effects: This invention converts multi-source, heterogeneous logs into standardized, structured data in real time. Standardized log data is stored in a search engine, and all subsequent analyses are based on unified semantic fields, masking underlying format differences. The engine can receive user-generated questions or fault descriptions in natural language and collaboratively execute them using a rule engine and a locally deployed lightweight local large-scale model. This eliminates the need for testers to memorize complex command-line syntax or query numerical user lines; they can perform efficient retrieval and preliminary analysis using natural language. An interactive guided interface integrates expert experience into the process, assisting novices in quickly locating problems. It breaks down information silos across logs and hosts, reducing manual correlation work that previously took hours to minutes. Combining the determinism of the rule engine with the semantic understanding and reasoning capabilities of the large language model, it can discover potential, indirect relationships, reducing false positives and false negatives. It upgrades traditional keyword retrieval to intelligent question answering and step-by-step guidance, thereby lowering the analysis threshold and significantly improving fault location efficiency.

[0048] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the log analysis method disclosed in any of the foregoing embodiments. Furthermore, the electronic device in this embodiment may specifically be an electronic computer.

[0049] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0050] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0051] The operating system 221 is used to manage and control the various hardware devices on the electronic device and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the log analysis method executed by the electronic device as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0052] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed log analysis method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0053] Furthermore, this application also discloses a computer program product, including a computer program / instructions; wherein, when the computer program / instructions are executed by a processor, they implement the aforementioned disclosed log analysis method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0055] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0057] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A log analysis method, characterized in that, include: The collected multi-source heterogeneous logs are converted into structured data in a preset format, and the structured data is stored in the search engine. The search engine receives user input information and parses the input information to obtain parsed keywords. The parsed keywords are matched with the structured data stored in the search engine to obtain the matched data; Based on the matched data, the local large model, and the preset log analysis instructions, the log analysis results are determined and displayed.

2. The log analysis method according to claim 1, characterized in that, Before converting the collected multi-source heterogeneous logs into structured data in a preset format, the process also includes: Deploy lightweight collectors on each server; Develop parsing plugins for each log type; the parsing plugins include preset regular expressions or delimiter rules; Multi-source heterogeneous logs are collected using the lightweight collector; Accordingly, the process of converting the collected multi-source heterogeneous logs into structured data in a preset format includes: The parsing plugin is invoked to parse the multi-source heterogeneous logs, and the corresponding parsing results are converted into structured data in a preset format. The preset format includes a timestamp, the IP address of the host that generated the log, the log source, the original log text, and the event type.

3. The log analysis method according to claim 1, characterized in that, Before storing the structured data in the search engine, the method further includes: The target deployment method is determined based on the amount of structured data; the target deployment method includes single-node deployment and minimum-scale cluster deployment. The search engine is deployed to the server based on the target deployment method.

4. The log analysis method according to claim 1, characterized in that, The parsed keywords include time, object, and event; Accordingly, the step of matching the parsed keywords with the structured data stored in the search engine to obtain the matched data includes: Based on the parsed keywords, time-based and fuzzy queries are performed on the structured data stored in the search engine to obtain the corresponding matched data.

5. The log analysis method according to claim 1, characterized in that, After matching the parsed keywords with the structured data stored in the search engine to obtain the matched data, the process further includes: The matched data is associated according to the timeline and preset rules to obtain associated data, which is then sent to the local large model through the search engine.

6. The log analysis method according to claim 5, characterized in that, The process of determining the log analysis result based on the matched data, the local large model, and preset log analysis instructions includes: The search engine sends the associated data and corresponding professional knowledge to the local large model, so that the local large model can output the corresponding log analysis results based on the associated data and preset log analysis instructions.

7. The log analysis method according to any one of claims 1 to 6, characterized in that, The presentation of the log analysis results includes: An answer summary is generated based on the log analysis results, and the answer summary is displayed. The log entries associated with the log analysis results are highlighted, and the order of occurrence of the events associated with the log analysis results is displayed in a chart format.

8. A log analysis device, characterized in that, include: The data storage module is used to convert the collected multi-source heterogeneous logs into structured data in a preset format, and store the structured data in the search engine; The parsing module is used to receive user input information through the search engine, parse the input information, and obtain parsed keywords; The data matching module is used to match the parsed keywords with the structured data stored in the search engine to obtain the matched data; The results display module is used to determine the log analysis results based on the matched data, the local large model, and the preset log analysis instructions, and to display the log analysis results.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the log analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the log analysis method as described in any one of claims 1 to 7.

Images