Intelligent log control method and system, computer and storage medium

Abstract

The invention provides an intelligent log control method and system, a computer and a storage medium. The method comprises the following steps: analyzing a text log and monitoring process state data; dynamically constructing a knowledge graph, and based on the knowledge graph, matching the current log event and the process state to a corresponding predefined operation instruction to generate a candidate instruction set; and calculating a priority score of each instruction in the candidate instruction set based on the node association strength and the real-time resource state of the knowledge graph, so as to sort the candidate instruction set and output the sorted candidate instruction set to an instruction execution engine. Multi-dimensional logs and state data are fused through a dynamic knowledge graph, context awareness and global optimization of instruction decisions are achieved, and misinformation and resource conflicts caused by isolated judgment in a traditional method are effectively avoided; a self-adaptive adjustment mechanism based on efficiency feedback is introduced, so that the system can continuously optimize an instruction priority strategy, the accuracy and efficiency of operation and maintenance response are remarkably improved, the manual intervention cost is finally reduced, and the system stability is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a log intelligent control method and system, a computer and a storage medium. BACKGROUND

[0002] The log data generated by a computer system contains data such as the running state of the system. In the operation and maintenance scenarios of cloud computing centers, large software systems and distributed server clusters, control instructions can be generated by analyzing logs, which can effectively improve the fault response speed, reduce the cost of manual intervention, and ensure the continuity of business services.

[0003] In the prior art, the log control method usually relies on a predefined rule library or a static threshold mechanism to trigger a predetermined operation instruction by matching log keywords or monitoring index thresholds, for example, automatically restarting the service process when the number of error logs exceeds the threshold. The rules and thresholds are usually preset by human experience, which is difficult to adapt to the dynamically changing operation and maintenance environment, resulting in a long-term decline in operation and maintenance efficiency and a high false alarm rate. SUMMARY

[0004] In view of the deficiencies of the prior art, the purpose of the present application is to provide a log intelligent control method and system, a computer and a storage medium, which aims to solve the technical problems of long-term decline in operation and maintenance efficiency and high false alarm rate in the prior art.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a log intelligent control method, comprising the following steps: Collecting multi-source log data, the log data including text logs and process states; Analyzing the text logs to extract log event types, levels and timestamp information; Monitoring the process state data to obtain CPU occupancy, memory occupancy and process running state in real time; Dynamically building a knowledge graph, wherein the nodes of the knowledge graph include log events extracted from the text logs, process states obtained from the process state data and predefined operation instructions, and the edge weights of the knowledge graph are calculated based on the correlation coefficient of log levels and process resource occupancy; Based on the knowledge graph, matching the current log event and process state to the corresponding predefined operation instruction to generate a candidate instruction set; Based on the node association strength of the knowledge graph and the real-time resource state, calculating the priority score of each instruction in the candidate instruction set to sort the candidate instruction set according to the priority score and generate a to-be-executed instruction list output to an instruction execution engine.

[0006] According to an aspect of the above technical solution, the edge weight of the knowledge graph is calculated by using a modularity optimization algorithm, and a calculation expression of the modularity parameter is: ; ; In the formula, is a modularity parameter, m is the sum of the weights of all edges in the graph, n is the total number of nodes, is the association strength between node i and node j, is the co-occurrence number of log time and process state in a time window, is the total number of events, is an event similarity judgment function, and a value of 1 indicates that nodes i and j are in the same event category, and a value of 0 indicates different event categories, is the sum of the weights of all adjacent edges connected to node i in the network, is the sum of the weights of all adjacent edges connected to node j in the network.

[0007] According to an aspect of the above technical solution, a calculation expression of the priority score is: ; In the formula, is the CPU occupancy rate, indicates the memory occupancy rate, is the weight value of the log level mapping extracted in the text log, , , is a dynamic adjustment coefficient.

[0008] According to an aspect of the above technical solution, the update of the dynamic adjustment coefficient is adjusted in sensitivity by using an efficiency value, and a calculation expression of the efficiency value is: ; In the formula, is the efficiency value, is the actual time consumption of the instruction, is a preset timeout threshold, is the number of error logs after execution of the instruction, is the total amount of logs generated during execution of the instruction.

[0009] According to an aspect of the above technical solution, a calculation expression of the adjusted dynamic adjustment coefficient is: ; In the formula, is the adjusted dynamic adjustment coefficient, is the coefficient value before iteration, is a learning rate, a difference between a current performance value and a last iteration performance value, a difference between a current coefficient value and a last iteration coefficient value.

[0010] According to an aspect of the above technical solution, the dynamic updating trigger condition of the knowledge graph is: when the addition frequency of the log event node exceeds a first threshold value, or the coefficient of variation of the process resource occupation rate exceeds a second threshold value.

[0011] In a second aspect, the present application provides a log intelligent control system, comprising: a collection module for collecting multi-source log data, the log data including text logs and process states; a parsing module for parsing the text logs to extract log event types, levels and timestamp information; a monitoring module for monitoring the process state data to obtain CPU occupation rate, memory occupation rate and process running state in real time; a construction module for dynamically constructing a knowledge graph, wherein the nodes of the knowledge graph include log events extracted from the text logs, process states obtained from the process state data and predefined operation instructions, and the edge weights of the knowledge graph are calculated based on the correlation coefficient between log levels and process resource occupation; a candidate module for matching the current log event and process state to the corresponding predefined operation instruction based on the knowledge graph to generate a candidate instruction set; a sorting module for calculating the priority score of each instruction in the candidate instruction set based on the node association strength of the knowledge graph and the real-time resource state, and sorting the candidate instruction set according to the priority score to generate a to-be-executed instruction list output to an instruction execution engine.

[0012] Compared with the prior art, the present application has the beneficial effects that by dynamically fusing multi-dimensional logs and state data through a knowledge graph, context awareness and global optimization of instruction decision-making are realized, effectively avoiding false alarms and resource conflicts caused by isolated judgment in traditional methods; at the same time, an adaptive adjustment mechanism based on performance feedback is introduced, so that the system can continuously optimize the instruction priority strategy, significantly improving the accuracy and efficiency of operation and maintenance response, ultimately reducing the cost of manual intervention and ensuring system stability. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 a flowchart of the log intelligent control method in the first embodiment of the present application; Figure 2 a block diagram of the log intelligent control system in the second embodiment of the present application; Figure 3 a hardware structure diagram of the computer in the third embodiment of the present application; The present application will be further described with reference to the following detailed description in connection with the above-mentioned drawings. DETAILED DESCRIPTION

[0014] For the purposes of this disclosure, reference will be made to the accompanying drawings which form a part of the specification. Several embodiments of the present application are described in the detailed description which follows. It is intended that all such embodiments come within the scope of the present application. It is also intended that individual features of the embodiments are interchangeable so that single features can be used in other embodiments, in combination with other features to produce yet other embodiments. It is intended that the present application embraces all such embodiments.

[0015] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be understood that the terms "approximately" and "substantially" are used herein to represent the insubstantial difference in the precision of a numerical value. It is to be understood that the terms "coupled" and "connected," along with derivatives thereof, are used herein to refer to any connection, coupling, or relation between or among two or more elements, and can be electrically, magnetically, or mechanically related, and that the relation between or among two or more elements includes the possibility of a direct electrical or magnetic connection between or among the two or more elements.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety for the teachings relevant to the sentence and / or paragraph in which the reference is presented.

[0017] Embodiment One Referring to Figure 1 , a flow chart of a log intelligent control method in a first embodiment of the present application is shown. As shown in the figure, the method includes the following steps: Step S100, collect multi-source log data, the log data including text log and process state. Real-time collection of text log (such as Syslog, JSON format application log) and process state data (CPU, memory, disk I / O indicators obtained through Linux proc file system or Prometheus exporter) is performed through a lightweight agent program deployed on a server node.

[0018] Step S200, parse the text log to extract log event type, level and timestamp information.

[0019] Specifically, in this embodiment, the parsing of the text log uses regular rule matching structured log for structured log such as error code, timestamp, etc.; and uses natural language processing model such as BERT fine-tuning model to identify key events for unstructured log, such as "Connection timeout to database" classified as DB connection event.

[0020] Step S300, dynamically constructing a knowledge graph, wherein the nodes of the knowledge graph include log events extracted from text logs, process states obtained from process state data, and predefined operation instructions, and the edge weights of the knowledge graph are calculated based on the correlation coefficients of log levels and process resource occupancy. The node types include event nodes (such as "CPU overload"), state nodes (such as "memory usage > 90%"), and instruction nodes (such as "restart service"), and the edge weights are calculated using the Pearson correlation coefficient formula.

[0021] Specifically, in this embodiment, the edge weights of the knowledge graph are calculated using a modularity optimization algorithm, and the calculation expression of the modularity parameter is: ; ; In the formula, is the modularity parameter, m is the sum of all edge weights in the graph, n is the total number of nodes, is the association strength between node i and node j, is the number of co-occurrences of log time and process state in the time window, is the total number of events, is an event similarity judgment function, taking a value of 1 to indicate that nodes i and j are of the same event category, and taking a value of 0 to indicate different event categories, is the sum of the weights of all adjacent edges connected to node i in the network, is the sum of the weights of all adjacent edges connected to node j in the network.

[0022] Step S400, based on the knowledge graph, matching the current log event and process state to the corresponding predefined operation instructions to generate a candidate instruction set. Specifically, the matching mechanism performs multi-hop queries based on the knowledge graph: 1. Take the current active event node as the starting point; 2. Traverse the instruction nodes directly connected thereto; 3. Calculate the total path weight, and if the path weight is greater than a preset value, add the treatment to the candidate set. When multiple instructions match the same event, retain the top 3 instructions with the highest path weight.

[0023] Step S500, calculating the priority score of each instruction in the candidate instruction set based on the node association strength of the knowledge graph and the real-time resource state, and sorting the candidate instruction set according to the priority score to generate a to-be-executed instruction list and output it to the instruction execution engine.

[0024] Further, in this embodiment, the calculation expression of the priority score is: ; In the formula, is the CPU occupancy rate, represents the memory occupancy rate, is the weight value of the log level mapping extracted in the text log, , , is a dynamic adjustment coefficient. is the weight of the CPU occupancy rate in the priority calculation, is the weight of the memory occupancy rate in the priority calculation, is the weight of the log level weight in the priority calculation. , , The sum of the above is 1.

[0025] Further, the update of the dynamic adjustment coefficient is sensitive to the performance value, and the calculation expression of the performance value is: ; In the formula, is the performance value, is the actual time consumption of the instruction, is a preset timeout threshold, is the number of error logs after the execution of the instruction, is the total amount of logs generated during the execution of the instruction.

[0026] The calculation expression of the adjusted dynamic adjustment coefficient is: ; In the formula, is the adjusted dynamic adjustment coefficient, is the coefficient value before iteration, is the learning rate, is the difference between the current performance value and the performance value of the last iteration, is the difference between the current coefficient value and the coefficient value of the last iteration. By dynamically adjusting the coefficients (such as CPU / memory weights) in the priority formula, the system is more suitable for real-time load changes. Understandably, , The adjustment calculation expression of the above is the same as the above, and will not be described in detail here.

[0027] Preferably, in the present embodiment, the dynamic update triggering condition of the knowledge graph is: When the addition frequency of the log event node exceeds a first threshold, or the coefficient of variation of the process resource occupancy rate exceeds a second threshold.

[0028] ​To sum up, the log intelligent control method in the above embodiments of the present application realizes context perception and global optimization of instruction decision by fusing multi-dimensional logs and state data through a dynamic knowledge graph, effectively avoiding false positives and resource conflicts caused by isolated judgment in traditional methods; meanwhile, an adaptive adjustment mechanism based on performance feedback is introduced, enabling the system to continuously optimize the instruction priority strategy, significantly improving the accuracy and efficiency of operation and maintenance response, and ultimately reducing the cost of manual intervention and ensuring system stability.

[0029] Embodiment Two The second embodiment of the present application also provides a log intelligent control system for implementing the embodiments and preferred embodiments, which have been described above and will not be repeated. As used below, the terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware implementation is also possible and contemplated.

[0030] As shown in the Figure 2 The system comprises a collection module 100, an analysis module 200, a monitoring module 300, a construction module 400, a candidate module 500 and a sorting module 600.

[0031] The collection module 100 is configured to collect multi-source log data, wherein the log data comprises text logs and process state data; The analysis module 200 is configured to analyze the text logs and extract log event types, levels and timestamps; The monitoring module 300 is configured to monitor the process state data and obtain CPU occupancy, memory occupancy and process running state in real time; The construction module 400 dynamically constructs a knowledge graph, wherein the nodes of the knowledge graph comprise log events extracted from the text logs, process states obtained from the process state data and predefined operation instructions, and the edge weights of the knowledge graph are calculated based on the correlation coefficient of log levels and process resource occupancy; The candidate module 500 is configured to match the current log events and process states to corresponding predefined operation instructions based on the knowledge graph, and generate a candidate instruction set; The sorting module 600 is configured to calculate the priority scores of the instructions in the candidate instruction set based on the node association strength of the knowledge graph and the real-time resource state, sort the candidate instruction set according to the priority scores, and generate a to-be-executed instruction list output to an instruction execution engine.

[0032] It should be noted that the various modules can be functional modules or program modules, and can be implemented by software or hardware. For the modules implemented by hardware, the various modules can be located in the same processor, or the various modules can also be located in different processors in any combination.

[0033] Embodiment three The third embodiment of the present application provides a computer, which can include a processor 81 and a memory 82 storing computer program commands.

[0034] Specifically, the processor 81 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0035] The memory 82 can include mass storage for data or commands. By way of example, and not limitation, memory 82 can include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, a USB drive, a magneto-optical disk, optical disk, a tape drive, or a combination of two or more of these. Storage 82 can be removable. Storage 82 can be internal or external. In some embodiments, storage 82 is non-volatile memory. In some embodiments, storage 82 includes read-only memory (ROM) and random-access memory (RAM). The ROM can be mask- programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or FLASH, or a combination of two or more of these, as appropriate. The RAM can be static RAM (SRAM) or dynamic RAM (DRAM), which can be Fast Page Mode DRAM (FPM DRAM), Extended Data Output DRAM (EDO DRAM), synchronous DRAM (SDRAM), etc., as appropriate.

[0036] The memory 82 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program commands executed by the processor 81.

[0037] The processor 81 reads and executes the computer program commands stored in the memory 82 to implement any one of the log intelligent control methods in the above embodiments.

[0038] In some embodiments, the computer can further include a communication interface 83 and a bus 80. In which, as shown, the processor 81, the memory 82, the communication interface 83 are connected through the bus 80 and complete the communication among each other. Figure 3

[0039] The communication interface 83 is used to realize the communication among the modules, devices, units and / or equipment in the embodiments of the present application. The communication interface 83 can also realize the data communication between other components, such as: external devices, image / data acquisition devices, databases, external storage and image / data processing workstations, etc.

[0040] ​Bus 80 includes hardware, software, or both, to couple components of computer 10 to each other and to couple computer 10 to other systems or devices. While bus 80 is shown in Figure 1 as a single bus, alternative embodiments include one or more buses. Bus 80 can be any of several types of bus structures including, but not limited to, a data bus, an address bus, a control bus, an expansion bus, a local bus, etc. In one embodiment, bus 80 includes a graphics accelerator interface (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or interconnect, or a combination of two or more of these. Where appropriate, bus 80 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0041] Embodiment Four A readable storage medium is provided in the fourth embodiment of the application. The readable storage medium stores computer program commands. The computer program commands are executed by a processor to implement any of the log intelligent control methods in the above embodiments.

[0042] Any combination of the above-described technical features of the embodiments can be made. To make the description simple, all possible combinations of the technical features in the embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0043] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a more specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A log intelligent control method, characterized in that, Includes the following steps: Collect multi-source log data, including text logs and process status; Parse the text log to extract log event type, level, and timestamp information; Monitor the process status data to obtain CPU utilization, memory utilization, and process running status in real time; A knowledge graph is dynamically constructed. The nodes of the knowledge graph include log events extracted from text logs, process states obtained from process state data, and predefined operation instructions. The edge weights of the knowledge graph are calculated based on the correlation coefficient between log level and process resource consumption. Based on the knowledge graph, the current log events and process states are matched to the corresponding predefined operation instructions to generate a candidate instruction set; Based on the node association strength and real-time resource status of the knowledge graph, the priority score of each instruction in the candidate instruction set is calculated, and the candidate instruction set is sorted according to the priority score to generate a list of instructions to be executed and output to the instruction execution engine.

2. The log intelligent control method according to claim 1, characterized in that, The edge weights of the knowledge graph are calculated using a modularity optimization algorithm. The expression for calculating the modularity parameter is as follows: ； ； In the formula, Here, m is the modularity parameter, m is the sum of the weights of all edges in the graph, and n is the total number of nodes. Let be the association strength between node i and node j. This represents the number of times the log time and process status co-occur within the time window. The total number of events, This is a function for determining event similarity. A value of 1 indicates that nodes i and j belong to the same event category, while a value of 0 indicates that they belong to different event categories. This is the sum of the weights of all adjacent edges connected to node i in the network. It is the sum of the weights of all adjacent edges connected to node j in the network.

3. The intelligent log control method according to claim 1, characterized in that, The expression for calculating the priority score is: ； In the formula, CPU utilization Indicates memory usage. The weight values ​​mapped to the log levels extracted from the text logs. , , This is a dynamically adjusted coefficient.

4. The intelligent log control method according to claim 3, characterized in that, The dynamic adjustment coefficient is updated based on sensitivity adjustment using the performance value, which is calculated using the following expression: ； In the formula, For performance value, The actual time taken for the instruction. The preset timeout threshold, The number of error log entries after the instruction is executed. This represents the total amount of logs generated during instruction execution.

5. The intelligent log control method according to claim 4, characterized in that, The formula for calculating the adjusted dynamic adjustment coefficient is as follows: ； In the formula, This is the adjusted dynamic adjustment coefficient. These are the coefficient values ​​before the iteration begins. For learning rate, This is the difference between the current performance value and the performance value of the previous iteration. This is the difference between the current coefficient value and the coefficient value of the previous iteration.

6. The intelligent log control method according to claim 1, characterized in that, The dynamic update trigger condition for the knowledge graph is: When the frequency of new log event nodes exceeds the first threshold, or the coefficient of variation of process resource utilization exceeds the second threshold.

7. A log intelligent control system, characterized in that, include: The acquisition module is used to collect multi-source log data, including text logs and process status. The parsing module is used to parse the text log and extract log event type, level, and timestamp information; The monitoring module is used to monitor the process status data and obtain CPU usage, memory usage and process running status in real time. The module dynamically constructs a knowledge graph, where the nodes of the knowledge graph include log events extracted from text logs, process states obtained from process state data, and predefined operation instructions. The edge weights of the knowledge graph are calculated based on the correlation coefficient between log level and process resource consumption. The candidate module is used to match the current log events and process states to corresponding predefined operation instructions based on the knowledge graph, and generate a candidate instruction set; The sorting module is used to calculate the priority score of each instruction in the candidate instruction set based on the node association strength and real-time resource status of the knowledge graph, so as to sort the candidate instruction set according to the priority score and generate a list of instructions to be executed and output it to the instruction execution engine.

8. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the log intelligent control method as described in any one of claims 1-6.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the log intelligent control method as described in any one of claims 1-6 above.

Images