Real-time log analysis method, system and equipment based on big data and storage medium
By using log file processing models for acquisition and clustering, combined with adjustments to the trained model, the problem of decreased fault identification accuracy caused by changes in server status was solved, achieving higher fault diagnosis accuracy.
Patent Information
- Application Number
- CN202310483354.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-04
- Publication Date
- 2024-11-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing technology cannot adapt to changes in server status, resulting in a decrease in the accuracy of identifying fault types represented by log files.
By acquiring feature information of historical and real-time log files through a log file processing model, clustering processing and model training are used to determine the target cluster to which the real-time log files belong. The model is trained when the inference error exceeds a threshold, and the weights of the loss function are adjusted to improve accuracy.
It improves the accuracy of fault diagnosis, making the characteristic information of log files more adaptable to changes in server status, and enabling more accurate identification of fault types.
Smart Images

Figure CN118897745A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method, system, device and storage medium for real-time log analysis based on big data. Background Art
[0003] In the related art, CN115794548A provides a method and device for detecting log anomalies, which includes: obtaining operation logs in real time and storing the operation logs in an operation log data set; parsing the operation logs to obtain an operation log template sequence; assigning a unique identifier to each operation log template to obtain a target log template sequence; extracting features from the target log template sequence to obtain a target feature vector sequence; inputting the target feature vector sequence into the detection model, and outputting the prediction result of the next operation log; the detection model is obtained by training an improved temporal convolutional neural network with a training sample set; the improved temporal convolutional neural network is composed of multiple residual blocks and a network adaptation layer; the residual block includes causal convolution and dilated convolution; the network adaptation layer includes a fully connected layer and a Softmax activation function. While the accuracy of log anomaly prediction is greatly improved, the time consumption is reduced.
[0004] CN115705413A provides a method and device for determining abnormal logs. The method includes: obtaining a real-time log sequence according to a first preset time length; inputting the real-time log sequence into a log quantity anomaly detection model, and performing quantity anomaly detection on the number of logs in the real-time log sequence; when the number of logs in the real-time log sequence is abnormal, inputting the real-time log sequence into a classification anomaly detection model, and performing feature anomaly detection on the real-time log sequence; when the result of the feature anomaly detection of the real-time log sequence is abnormal, inputting the log data of the second preset time length before the current moment into a time series anomaly detection model to judge the number distribution of log patterns, and determining abnormal logs. The method and device for determining abnormal logs provided by the present invention improve the detection speed and accuracy of logs by first determining a real-time log sequence with an abnormal number, and then detecting abnormal logs according to a classification anomaly detection model and a time series anomaly detection model.
[0005] CN115658360A provides a cloud system fault diagnosis method based on log data. The method includes the following steps: preprocessing the input log data, extracting log templates, timestamps and fingerprint sets; caching the preprocessed data, sorting the data in the cache by timestamp and extracting additional fingerprint sets; dividing discrete logs by tasks to form workflows; generating references with task tags, and determining the task category to which the workflow belongs by the similarity between the generated workflow and the reference; building a diagnostic automaton for each task based on an automaton model; calculating the word frequency-inverse file frequency value of each log template in the training data set to obtain a word frequency-inverse file frequency dictionary; using the word frequency-inverse file frequency dictionary to filter real-time log data, and completing fault diagnosis and location by matching the filtered log sequence with the diagnostic automaton.
[0006] Therefore, in the relevant technology, system or equipment faults can be detected and diagnosed based on log files. However, as log files are continuously generated and the operating status of the system or equipment continues to change, the type of fault represented by the log file may change, and the relevant technology cannot adapt to such changes, which may cause the accuracy of fault identification to continue to decline over time.
[0007] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0009] The embodiments of the present invention provide a method, system, device and storage medium based on big data real-time log analysis, which can adapt to the continuous changes in server status and improve the accuracy of fault inference.
[0010] A first aspect of an embodiment of the present invention provides a method for real-time log analysis based on big data, the method being used in a server and comprising: Obtain historical feature information of historical log files of the server through the log file processing model; Clustering the historical feature information and the historical status information of the server corresponding to the historical log file to obtain a plurality of historical cluster centers and historical cluster clusters, wherein the historical status information is the historical running status of the server at the time when the historical log file is generated; When a real-time log file is obtained, first characteristic information of the real-time log file and a first operating state of the server are obtained through the log file processing model; Determine a target cluster to which the real-time log file belongs according to the first feature information of the real-time log file, the first operating state, and the plurality of historical cluster centers; Obtaining, by means of the log file processing model, inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs; Determining an inference error based on the inferred feature information and the target cluster center of the target cluster; When the inferred error is greater than or equal to a preset error threshold, the log file processing model is trained using the inferred feature information and the target cluster center to obtain a trained log file processing model.
[0011] According to an embodiment of the present invention, obtaining the inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs through the log file processing model includes: Obtaining first state characteristic information according to the first characteristic information and the first operating state; Obtaining historical status feature information according to the historical feature information and historical status information in the target cluster; Obtaining a state distance matrix according to the first state feature information and the historical state feature information; Determine a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center; The first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed through the log file processing model to obtain the inferred feature information.
[0012] According to an embodiment of the present invention, obtaining a state distance matrix according to the first state feature information and the historical state feature information includes: According to the formula , determine the state distance matrix D s , where s1 is the first state characteristic information, is the i-th historical state feature information, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0013] According to an embodiment of the present invention, determining a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center includes: According to the formula Determine the center distance matrix Do , where s1 is the first state characteristic information, is the i-th historical state feature information, o is the target cluster center, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0014] According to an embodiment of the present invention, the first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed by the log file processing model to obtain the inferred feature information, including: According to the formula , obtain the output information OUT1 of the first processing level of the log file processing model, where D s is the state distance matrix, D o is the center distance matrix, I1 is the first state feature information, I h is the historical state feature information, L1 is the parameter matrix of the first processing level of the log file processing model; According to the formula , obtain the output information OUT of the jth processing level of the log file processing model j , where j is a positive integer greater than 1, L j is the parameter matrix of the jth processing level of the log file processing model; The inferred feature information is obtained according to output information of the last processing level of the log file processing model.
[0015] According to an embodiment of the present invention, when the inference error is greater than or equal to a preset error threshold, training the log file processing model by using the inferred feature information and the target cluster center to obtain the trained log file processing model includes: According to the formula Get the loss function Loss; where o p is the inferred feature information, o is the target cluster center, The first state characteristic information I1 and the historical state characteristic information I h The column vector obtained by weighted averaging the column vectors of the matrix composed of The output information OUT of the kth processing level of the log file processing model k The column vector obtained by weighted average of multiple column vectors of α, β k , γ and μ are all preset weights, and the preset weights meet T is the number of historical cluster centers, ot is the t-th historical cluster center, t is a positive integer and t≤T; According to the loss function Loss, the log file processing model is trained to obtain a trained log file processing model.
[0016] According to one embodiment of the present invention, the method further includes: Re-obtaining historical feature information of historical log files and first feature information of real-time log files through the trained log file processing model; According to the re-acquired historical feature information, the re-acquired first feature information, the historical state information and the first operating state, an adjusted historical cluster center and an adjusted historical cluster cluster are obtained.
[0017] According to a second aspect of an embodiment of the present invention, there is provided a real-time log analysis system based on big data, the system being arranged on a server and comprising: The history module is used to obtain the historical feature information of the server's historical log files through the log file processing model; A clustering module, used for clustering the historical feature information and the historical status information of the server corresponding to the historical log file to obtain a plurality of historical clustering centers and historical clustering clusters, wherein the historical status information is the historical running status of the server at the time when the historical log file is generated; A real-time log module, configured to obtain, when a real-time log file is obtained, first characteristic information of the real-time log file and a first operating state of the server through the log file processing model; A target cluster module, used to determine the target cluster to which the real-time log file belongs according to the first characteristic information of the real-time log file, the first operating state and the multiple historical cluster centers; an inference module, configured to obtain inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs, through the log file processing model; An error module, used to determine an inference error based on the inferred feature information and the target cluster center of the target cluster; A training module is used to train the log file processing model through the inferred feature information and the target cluster center to obtain a trained log file processing model when the inferred error is greater than or equal to a preset error threshold.
[0018] According to one embodiment of the present invention, the inference module is further used for: Obtaining first state characteristic information according to the first characteristic information and the first operating state; Obtaining historical status feature information according to the historical feature information and historical status information in the target cluster; Obtaining a state distance matrix according to the first state feature information and the historical state feature information; Determine a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center; The first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed through the log file processing model to obtain the inferred feature information.
[0019] According to one embodiment of the present invention, the inference module is further used for: According to the formula , determine the state distance matrix D s , where s1 is the first state characteristic information, is the i-th historical state feature information, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0020] According to one embodiment of the present invention, the inference module is further used for: According to the formula Determine the center distance matrix D o , where s1 is the first state characteristic information, is the i-th historical state feature information, o is the target cluster center, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0021] According to one embodiment of the present invention, the inference module is further used for: According to the formula , obtain the output information OUT1 of the first processing level of the log file processing model, where D s is the state distance matrix, D o is the center distance matrix, I1 is the first state feature information, I h is the historical state feature information, L1 is the parameter matrix of the first processing level of the log file processing model; According to the formula , obtain the output information OUT of the jth processing level of the log file processing model j , where j is a positive integer greater than 1, L j is the parameter matrix of the jth processing level of the log file processing model; The inferred feature information is obtained according to output information of the last processing level of the log file processing model.
[0022] According to one embodiment of the present invention, the training module is further used for: According to the formula Get the loss function Loss; where o p is the inferred feature information, o is the target cluster center, The first state characteristic information I1 and the historical state characteristic information I h The column vector obtained by weighted averaging the column vectors of the matrix composed of The output information OUT of the kth processing level of the log file processing model k The column vector obtained by weighted average of multiple column vectors of α, β k , γ and μ are all preset weights, and the preset weights meet T is the number of historical cluster centers, o t is the t-th historical cluster center, t is a positive integer and t≤T; According to the loss function Loss, the log file processing model is trained to obtain a trained log file processing model.
[0023] According to one embodiment of the present invention, the system further includes an adjustment module, which is used to: Re-obtaining historical feature information of historical log files and first feature information of real-time log files through the trained log file processing model; According to the re-acquired historical feature information, the re-acquired first feature information, the historical state information and the first operating state, an adjusted historical cluster center and an adjusted historical cluster cluster are obtained.
[0024] According to a third aspect of an embodiment of the present invention, a device for real-time log analysis based on big data is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the method for real-time log analysis based on big data.
[0025] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method for real-time log analysis based on big data.
[0026] According to the big data-based real-time log analysis method of an embodiment of the present invention, historical log files and historical state information can be clustered, and the target cluster to which the real-time log file belongs can be determined based on the historical cluster center, and the inferred feature information can be obtained through the log file processing model. In the inference process, the distance relationship between the first state feature and the historical state feature information, as well as the distance relationship between the first state feature and the historical state feature information and the target cluster center can be fully utilized to improve the inference accuracy. When the inference error between the target cluster center and the inferred feature information is greater than or equal to the preset error threshold, the log file processing model is trained, and during the training process, the weights of each item in the loss function are adjusted in real time according to the correctness of the classification of the inferred feature information to improve the training accuracy. The characteristic information of log files obtained using the trained log file processing model can better adapt to the state changes of the server, and can obtain more accurate characteristic information of the log files, and match the characteristic information of the log files to more accurate clustering clusters, thereby continuously improving the accuracy of fault diagnosis. In addition, the trained log file processing model can also be used to update the clustering clusters of historical log files, so that the characteristic information of historical log files can better adapt to the state changes of the server, that is, the fault type represented by the historical log file is more matched with the current state of the server, so that the fault type represented by the log file can be more accurately identified, thereby improving the accuracy of fault diagnosis.
[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work. Figure 1 The following is a flow chart showing a method for real-time log analysis based on big data according to an embodiment of the present invention. Figure 2 The block diagram of the big data real-time log analysis system according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0033] Figure 1 The following is a flow chart showing a method for real-time log analysis based on big data according to an embodiment of the present invention. Figure 1 As shown, the method includes: In step S101, historical feature information of historical log files of the server is obtained through the log file processing model; In step S102, clustering is performed on the historical feature information and the historical status information of the server corresponding to the historical log file to obtain a plurality of historical cluster centers and historical cluster clusters, wherein the historical status information is the historical operating status of the server at the time when the historical log file is generated; In step S103, when a real-time log file is obtained, first characteristic information of the real-time log file and a first operating state of the server are obtained through the log file processing model; In step S104, a target cluster to which the real-time log file belongs is determined according to the first characteristic information of the real-time log file, the first operating state, and the plurality of historical cluster centers; In step S105, the inferred feature information of the real-time log file is obtained through the log file processing model according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs; In step S106, determining an inference error based on the inferred feature information and the target cluster center of the target cluster; In step S107, when the inferred error is greater than or equal to a preset error threshold, the log file processing model is trained using the inferred feature information and the target cluster center to obtain a trained log file processing model.
[0034] According to the big data-based real-time log analysis method of an embodiment of the present invention, historical log files and historical status information can be clustered, and the target clustering cluster to which the real-time log file belongs can be determined based on the historical clustering center, and the inferred feature information can be obtained through the log file processing model, so that when the inferred error between the target clustering center and the inferred feature information is greater than or equal to the preset error threshold, the log file processing model is trained, and the feature information of the log file obtained using the trained log file processing model can better adapt to the state changes of the server, and more accurate feature information of the log file can be obtained, and the feature information of the log file can be matched to a more accurate clustering cluster, thereby continuously improving the accuracy of fault diagnosis, and the trained log file processing model can also be used to update the clustering cluster of the historical log file, so that the feature information of the historical log file is more adapted to the state changes of the server, that is, the fault type represented by the historical log file is more matched with the current state of the server, so that the fault type represented by the log file can be more accurately identified, thereby improving the accuracy of fault diagnosis.
[0035] According to one embodiment of the present invention, in step S101, the server may generate a large number of log files, i.e., historical log files, during the historical operation process, and the historical feature information of these historical log files may be obtained through the encoding level of the log file processing model. In the example, the historical feature information may be information in the form of vectors, or information in other forms such as matrices and tensors. The present invention does not limit the form of the historical feature information. The log file processing model may be a deep learning neural network model, such as a convolutional neural network model or a recurrent neural network model, etc. The present invention does not limit the type of the log file processing model.
[0036] According to an embodiment of the present invention, the historical log file may be a historical log file within a period of time, for example, within one month before the current time, within one week before the current time, etc. Of course, it may also be all historical log files generated since the server was put into operation or a certain task was put into operation. The present invention does not limit this.
[0037] According to one embodiment of the present invention, in step S102, clustering processing may be performed on the historical feature information and the historical status information of the server corresponding to the historical log file. In the example, the historical status information may be used to represent the historical operating status of the server at the time when the historical log file is generated, for example, the fault status of the server at that time, the traffic value, the number of visits received, the number of the task being executed, etc. The present invention does not limit the type of information included in the historical operating status. In the example, during the clustering processing, the historical feature information and the corresponding historical status information may be merged, and the merged information may be clustered. For example, the historical feature information is information in the form of a vector, and the historical status information may also form information in the form of a vector. The two may be merged to form a higher dimensional vector, thereby performing clustering processing, and multiple historical clustering clusters may be obtained. Each historical clustering cluster may have a historical clustering center, and each historical clustering cluster may include one or more of the above merged information.
[0038] According to one embodiment of the present invention, in step S103, when a real-time log file is received at the current moment, the first characteristic information of the real-time log file can be extracted through the coding layer of the log file processing model, and the operating state of the server at the current moment, that is, the first operating state, is collected at the same time. The form of the first characteristic information is the same as the form of the historical characteristic information, and the type of information included in the first operating state is the same as the type of information included in the historical operating state. The real-time log file can represent the current fault state of the server, and can also be used to infer the current fault type. Of course, a real-time log file can also be generated when there is no fault. The present invention does not limit the conditions for generating a real-time log file.
[0039] According to one embodiment of the present invention, in step S104, the target cluster described in the real-time log file can be determined, that is, to which historical cluster the real-time log file should belong. In the example, the information after the first characteristic information and the first operating state are merged can be used to compare with each historical cluster center, for example, the characteristic distance (for example, Euclidean distance) or similarity (for example, cosine similarity) between the merged information and each historical cluster center is solved, and the historical cluster center with the shortest characteristic distance or the highest similarity can be determined as the target cluster center, and the historical cluster corresponding to the target cluster center is the target cluster described in the real-time log file. The fault state of the server when the multiple historical log files in the target cluster are generated can be the same or similar to the fault state of the server when the real-time log file is generated, and the fault type of the server when the real-time log file is generated can be inferred based on this.
[0040] According to one embodiment of the present invention, as time goes by, it may happen that although the real-time log file has a high similarity with the historical log file in a certain historical clustering cluster, or has a high similarity with the historical clustering center, the cause of the fault has changed. For example, as time goes by, the components of the server age, so the hardware status of the server is inconsistent with other moments in history, resulting in the current cause of the fault being inconsistent with the cause of the fault when the historical log file was generated, or, as time goes by, the tasks performed by the server or the user group accessing the server changes, resulting in the current cause of the fault being inconsistent with the cause of the fault when the historical log file was generated. Therefore, there may be a deviation in inferring the fault type of the server when the real-time log file is generated based on the historical clustering cluster of the historical log file.
[0041] According to one embodiment of the present invention, in order to overcome the above-mentioned problem, a log processing model can be used to obtain inferred feature information based on real-time log files, and based on the inferred feature information, it can be determined whether the historical clustering clusters and historical clustering centers deviate significantly from the current state of the server, that is, it is determined whether the historical clustering clusters and historical clustering centers based on historical log files are still suitable for inferring the fault type of the server when the real-time log files are generated. If it is no longer applicable, the historical feature information of the historical log files can be re-extracted through the trained log processing model, and re-clustered to obtain adjusted historical clustering centers and adjusted historical clustering clusters suitable for inferring the current fault type. In addition, the feature information of the current real-time log file extracted using the trained log processing model is also more suitable for expressing the current fault state of the server.
[0042] According to an embodiment of the present invention, in step S105, the first characteristic information, the first operating state and the target cluster can be processed by the log file processing model to obtain the inferred characteristic information of the real-time log file. Step S105 includes: obtaining the first state characteristic information according to the first characteristic information and the first operating state; obtaining the historical state characteristic information according to the historical characteristic information and the historical state information in the target cluster; obtaining the state distance matrix according to the first state characteristic information and the historical state characteristic information; determining the center distance matrix according to the first state characteristic information, the historical state characteristic information and the target cluster center; processing the first state characteristic information, the historical state characteristic information, the state distance matrix and the center distance matrix by the log file processing model to obtain the inferred characteristic information.
[0043] According to one embodiment of the present invention, as described above, the first feature information and the first operating state may be merged, for example, merged into vector information with a higher dimension, to obtain first state feature information. The historical feature information in the target cluster and the corresponding historical state information may be merged, for example, merged into vector information with a higher dimension, to obtain historical state feature information.
[0044] According to an embodiment of the present invention, obtaining a state distance matrix according to the first state feature information and the historical state feature information includes: determining the state distance matrix D according to formula (1): s : (1) Wherein, s1 is the first state characteristic information, is the i-th historical state feature information, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0045] According to one embodiment of the present invention, in the state distance matrix, each element of the first row is the binary norm of the difference between each historical state feature information and the vector of the first state feature information, wherein the binary norm of the difference between the first state feature information and its own vector is replaced by 1 in order not to generate all-0 vectors during the calculation process. Each element of the second row is the binary norm of the difference between the first state feature information and each historical state feature information and the vector of the first historical state feature information, wherein the binary norm of the difference between the first historical state feature information and its own vector is replaced by 1 in order not to generate all-0 vectors during the calculation process. Each element of the i+1th row is the binary norm of the difference between the first state feature information and each historical state feature information and the vector of the i-th historical state feature information, wherein the binary norm of the difference between the i-th historical state feature information and its own vector is replaced by 1 in order not to generate all-0 vectors during the calculation process. Finally, a matrix including (m+1)×(m+1) elements is obtained. And because the second norm of the vector difference is a value greater than or equal to 0, all elements of the state distance matrix are greater than or equal to 0 and it is a symmetric matrix.
[0046] According to an embodiment of the present invention, determining a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center includes: determining the center distance matrix D according to formula (2): o : (2) Wherein, s1 is the first state characteristic information, is the i-th historical state feature information, o is the target cluster center, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0047] According to one embodiment of the present invention, in the center distance matrix, each element of the first row is respectively the ratio of the second norm of the vector difference between each historical state feature information and the target cluster center, the maximum value of the second norm of the vector difference between the first state feature information and the target cluster center, and the second norm of the vector difference between the i-th historical state feature information and the first state feature information, wherein, in order to make the denominator not 0, the element corresponding to the first state feature information itself, that is, the element of the first row and the first column is set to 1. Each element of the i+1th row is respectively the second norm of the vector difference between the first state feature information or each historical state feature information and the target cluster center, the maximum value of the second norm of the vector difference between the i-th historical state feature information and the target cluster center, and the second norm of the vector difference between each historical state feature information and the i-th historical state feature information, wherein, in order to make the denominator not 0, the element corresponding to the i-th historical state feature information, that is, the element of the i+1th row and the i+1th column is set to 1. According to an embodiment of the present invention, after obtaining the above state distance matrix and center distance matrix, the information of the input log file processing model (i.e., the first state feature information and the historical state feature information) can be processed by graph convolution to make full use of the information contained in each feature information. And the input information is optimized by graph convolution to reduce the calculation amount of the log file processing model and improve the accuracy of the log file processing model.
[0048] According to an embodiment of the present invention, the first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed by the log file processing model to obtain the inferred feature information, including: obtaining output information OUT1 of the first processing level of the log file processing model according to formula (3): (3) Among them, D s is the state distance matrix, D o is the center distance matrix, I1 is the first state feature information, I h is the historical state feature information, L1 is the parameter matrix of the first processing level of the log file processing model; According to formula (4), the output information OUT of the jth processing level of the log file processing model is obtained: j , (4) Where j is a positive integer greater than 1, L j is the parameter matrix of the jth processing level of the log file processing model; The inferred feature information is obtained according to output information of the last processing level of the log file processing model.
[0049] According to one embodiment of the present invention, the encoding level of the log file processing model can be used to extract historical feature information and first feature information. After the encoding level, the log file processing model may also include a processing level that can be used to input and process first state feature information and historical state feature information.
[0050] According to one embodiment of the present invention, as described above, the first state feature information and the historical state feature information can be processed by graph convolution, that is, using By multiplying the first state feature information and the historical state feature information, the distance relationship between the first state feature and the historical state feature information, as well as the distance relationship between the first state feature and the historical state feature information and the target cluster center, the log file processing model outputs more accurate inferred feature information. The output information of the first processing level can be obtained by multiplying the result of the multiplication with the parameter matrix of the first processing level.
[0051] According to one embodiment of the present invention, the subsequent multiple processing levels can multiply the output information of the previous level with the parameter matrix of the current level during processing, and can be combined with Subtract from each other to obtain the output information of this level. By subtracting the input information of the first processing level from the multiplication above to obtain the residual information as the output information of this level, the optimized input information can be referred to in each processing level, thereby continuously strengthening the distance relationship between the first state feature and the historical state feature information, as well as the distance relationship between the first state feature and the historical state feature information and the target cluster center, so as to make full use of these two relationships and obtain more accurate inferred feature information.
[0052] According to an embodiment of the present invention, the output information of each level of the log file processing model is information in matrix form, and the column vector of the output information of the last processing level can be weighted averaged to obtain the inferred feature information.
[0053] According to one embodiment of the present invention, in step S106, the inferred feature information can be used to characterize the fault type of the server, the target cluster center can be used to characterize the fault type of the server represented by the cluster cluster, and the inferred feature information is obtained based on the real-time log file and the historical log file obtained at the current moment, and is more suitable for the current fault type inference. Therefore, the inferred feature information is more representative than the target cluster center, that is, the inferred feature information can be used as the representative of the cluster cluster. The inferred error between the inferred feature information and the target cluster center can be determined. If the error is not significant, that is, less than the preset error threshold, there is no need to retrain the log file processing model and obtain new feature information. In other words, the target cluster center and the inferred feature information are not very different and can still be applied to the current fault type inference. However, if the inferred error is greater than or equal to the preset error threshold, it means that the target cluster center is no longer suitable for the current fault type inference, and it is necessary to retrain the log file processing model, re-extract the feature information and re-cluster to obtain a new cluster center.
[0054] According to an embodiment of the present invention, step S106 may include: according to formula (5) (5) Get the loss function Loss; where o p is the inferred feature information, o is the target cluster center, The first state characteristic information I1 and the historical state characteristic information I h The column vector obtained by weighted averaging the column vectors of the matrix composed of The output information OUT of the kth processing level of the log file processing model k The column vector obtained by weighted average of multiple column vectors of α, β k , γ and μ are all preset weights, and the preset weights conform to the following formula (6) (6) T is the number of historical cluster centers, o t is the t-th historical cluster center, t is a positive integer and t≤T; According to the loss function Loss, the log file processing model is trained to obtain a trained log file processing model.
[0055] According to an embodiment of the present invention, the first term of formula (5) can represent input error information, that is, the vector difference between the inferred feature information and the target cluster center, and the first state feature information I1 and the historical state feature information I hThe product of the column vector obtained by weighted averaging the column vectors of the matrix composed of the first state feature information I1 and the vector difference between the target cluster center and the target cluster center can be used to reduce the error between the inferred feature information and the target cluster center during the training process. h The error between the column vector obtained by weighted averaging the column vectors of the composed matrix and the target cluster center is reduced, so that after adding the first state feature information, the cluster clusters obtained by re-clustering are more compact, the intra-class distance is smaller, the inter-class distance is larger, the classification is clearer, and the inference accuracy is improved.
[0056] According to an embodiment of the present invention, the second item of formula (5) can represent the vector difference between the inferred feature information and the target cluster center, the product of the column vector obtained by weighted averaging multiple column vectors of the output information of each processing level of the log file processing model and the vector difference of the target cluster center, and the product is summed. During the training process, in addition to reducing the error between the inferred feature information and the target cluster center, the output of each processing level can also be made closer to the target cluster center, thereby improving the accuracy of the log file processing model.
[0057] According to an embodiment of the present invention, the third item of formula (5) can represent the second norm of the vector difference between the inferred feature information and the target cluster center. During the training process, the error between the two can be reduced, so that the cluster center obtained by re-clustering is more suitable for the current server fault type inference.
[0058] According to an embodiment of the present invention, the fourth item of formula (5) can represent the similarity between the inferred feature information and the target cluster center. During the training process, the similarity between the two can be made higher, so that the inferred feature information is closer to the cluster center, thereby making the distance between classes larger and the classification clearer, thereby improving the accuracy of fault inference.
[0059] According to an embodiment of the present invention, the above four items can be weighted and summed, and the weights meet the formula (6), that is, when the binary norm of the vector difference between the inferred feature information and the target cluster center is the minimum value among the binary norms of the vector differences between the inferred feature information and each historical cluster center, it indicates that the classification of the inferred feature information is correct. Therefore, during the training process, it is possible to focus on making the intra-class distance smaller, that is, making the weights of the first three items greater than the weight of the fourth item. If the binary norm of the vector difference between the inferred feature information and the target cluster center is not the minimum value among the binary norms of the vector differences between the inferred feature information and each historical cluster center, it indicates that the classification of the inferred feature information is wrong, that is, it is closer to other historical cluster centers and is classified into other cluster clusters. At this time, due to the classification error, the training intensity that makes the inter-class distance larger should be increased, that is, the weight of the fourth item should be larger to reduce the probability of classification error.
[0060] According to one embodiment of the present invention, iterative training can be performed multiple times, for example, using the inferred feature information of multiple real-time log files and historical log files to perform the above training until the loss function converges, and a trained log file processing model can be obtained. The feature information of the log files extracted by the trained log file processing model is more suitable for inferring the fault type of the server in the current state.
[0061] According to one embodiment of the present invention, the method further includes: reacquiring historical feature information of historical log files and first feature information of real-time log files through a trained log file processing model; obtaining an adjusted historical clustering center and an adjusted historical clustering cluster according to the reacquired historical feature information, the reacquired first feature information, the historical state information and the first operating state. That is, re-extracting the first feature information and the historical feature information through the trained log file processing model, and clustering the first feature information and its corresponding first operating state, as well as the historical feature information and its corresponding historical state information, to obtain an adjusted historical clustering center and an adjusted historical clustering cluster. The adjusted historical clustering center and the adjusted historical clustering cluster are more suitable for inferring the fault type of the server in the current state.
[0062] According to one embodiment of the present invention, the processing of the inferred error between the above test inferred feature information and the target cluster center can be run once every preset time period, for example, every month, every quarter, every half year or every year, etc. The present disclosure does not limit the interval time period. The above operation can make the historical cluster center and the historical cluster cluster always applicable to the fault inference of the server, and make the first feature information of the real-time log file extracted by the log file processing model always applicable to the fault inference of the server. In the interval time period, the historical cluster cluster to which the first feature information belongs can be directly used for fault inference, without the need to use the log file processing model for subsequent processing, so as to save computing resources and improve the real-time performance of inferring faults through log files.
[0063] According to the big data-based real-time log analysis method of an embodiment of the present invention, historical log files and historical state information can be clustered, and the target cluster to which the real-time log file belongs can be determined based on the historical cluster center, and the inferred feature information can be obtained through the log file processing model. In the inference process, the distance relationship between the first state feature and the historical state feature information, as well as the distance relationship between the first state feature and the historical state feature information and the target cluster center can be fully utilized to improve the inference accuracy. When the inference error between the target cluster center and the inferred feature information is greater than or equal to the preset error threshold, the log file processing model is trained, and during the training process, the weights of each item in the loss function are adjusted in real time according to the correctness of the classification of the inferred feature information to improve the training accuracy. The characteristic information of log files obtained using the trained log file processing model can better adapt to the state changes of the server, and can obtain more accurate characteristic information of the log files, and match the characteristic information of the log files to more accurate clustering clusters, thereby continuously improving the accuracy of fault diagnosis. In addition, the trained log file processing model can also be used to update the clustering clusters of historical log files, so that the characteristic information of historical log files can better adapt to the state changes of the server, that is, the fault type represented by the historical log file is more matched with the current state of the server, so that the fault type represented by the log file can be more accurately identified, thereby improving the accuracy of fault diagnosis.
[0064] Figure 2 A block diagram of a real-time log analysis system based on big data according to an embodiment of the present invention is shown as follows: Figure 2 As shown, the system is arranged on a server and includes: The history module 101 is used to obtain historical feature information of the historical log files of the server through the log file processing model; A clustering module 102, configured to perform clustering processing on the historical feature information and the historical status information of the server corresponding to the historical log file, to obtain a plurality of historical clustering centers and historical clustering clusters, wherein the historical status information is the historical operating status of the server at the time when the historical log file is generated; The real-time log module 103 is used to obtain the first characteristic information of the real-time log file and the first operation state of the server through the log file processing model when the real-time log file is obtained; A target cluster module 104, configured to determine a target cluster to which the real-time log file belongs according to the first characteristic information of the real-time log file, the first operating state, and the plurality of historical cluster centers; An inference module 105, configured to obtain inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs, through the log file processing model; An error module 106, configured to determine an inference error based on the inferred feature information and the target cluster center of the target cluster; The training module 107 is used to train the log file processing model through the inferred feature information and the target cluster center to obtain a trained log file processing model when the inferred error is greater than or equal to a preset error threshold.
[0065] According to one embodiment of the present invention, the inference module is further used for: Obtaining first state characteristic information according to the first characteristic information and the first operating state; Obtaining historical status feature information according to the historical feature information and historical status information in the target cluster; Obtaining a state distance matrix according to the first state feature information and the historical state feature information; Determine a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center; The first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed through the log file processing model to obtain the inferred feature information.
[0066] According to one embodiment of the present invention, the inference module is further used for: According to the formula , determine the state distance matrix D s , where s1 is the first state characteristic information, is the i-th historical state feature information, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0067] According to one embodiment of the present invention, the inference module is further used for: According to the formula Determine the center distance matrix D o , where s1 is the first state characteristic information, is the i-th historical state feature information, o is the target cluster center, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
[0068] According to one embodiment of the present invention, the inference module is further used for: According to the formula , obtain the output information OUT1 of the first processing level of the log file processing model, where D s is the state distance matrix, Do is the center distance matrix, I1 is the first state feature information, I h is the historical state feature information, L1 is the parameter matrix of the first processing level of the log file processing model; According to the formula , obtain the output information OUT of the jth processing level of the log file processing model j , where j is a positive integer greater than 1, L j is the parameter matrix of the jth processing level of the log file processing model; The inferred feature information is obtained according to output information of the last processing level of the log file processing model.
[0069] According to one embodiment of the present invention, the training module is further used for: According to the formula Get the loss function Loss; where o p is the inferred feature information, o is the target cluster center, The first state characteristic information I1 and the historical state characteristic information I h The column vector obtained by weighted averaging the column vectors of the matrix composed of The output information OUT of the kth processing level of the log file processing model k The column vector obtained by weighted average of multiple column vectors of α, β k , γ and μ are all preset weights, and the preset weights meet T is the number of historical cluster centers, o t is the t-th historical cluster center, t is a positive integer and t≤T; According to the loss function Loss, the log file processing model is trained to obtain a trained log file processing model.
[0070] According to one embodiment of the present invention, the system further includes an adjustment module, which is used to: Re-obtaining historical feature information of historical log files and first feature information of real-time log files through the trained log file processing model; According to the re-acquired historical feature information, the re-acquired first feature information, the historical state information and the first operating state, an adjusted historical cluster center and an adjusted historical cluster cluster are obtained.
[0071] According to one embodiment of the present invention, there is provided a device for real-time log analysis based on big data, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the real-time log analysis method based on big data.
[0072] According to one embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the real-time log analysis method based on big data is implemented.
[0073] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.
[0074] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A real-time log analysis method based on big data, characterized in that: The method is used in a server and comprises: Obtain historical feature information of historical log files of the server through the log file processing model; Clustering the historical feature information and the historical status information of the server corresponding to the historical log file to obtain a plurality of historical cluster centers and historical cluster clusters, wherein the historical status information is the historical running status of the server at the time when the historical log file is generated; When a real-time log file is obtained, first characteristic information of the real-time log file and a first operating state of the server are obtained through the log file processing model; Determine a target cluster to which the real-time log file belongs according to the first feature information of the real-time log file, the first operating state, and the plurality of historical cluster centers; Obtaining, by means of the log file processing model, inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs; Determining an inference error based on the inferred feature information and the target cluster center of the target cluster; When the inferred error is greater than or equal to a preset error threshold, the log file processing model is trained by using the inferred feature information and the target cluster center to obtain a trained log file processing model.
2. The method for real-time log analysis based on big data according to claim 1 is characterized in that: Obtaining, by means of the log file processing model, inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs, includes: Obtaining first state characteristic information according to the first characteristic information and the first operating state; Obtaining historical status feature information according to the historical feature information and historical status information in the target cluster; Obtaining a state distance matrix according to the first state feature information and the historical state feature information; Determine a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center; The first state feature information, the historical state feature information, the state distance matrix and the center distance matrix are processed through the log file processing model to obtain the inferred feature information.
3. The method for real-time log analysis based on big data according to claim 2 is characterized in that: Obtaining a state distance matrix according to the first state feature information and the historical state feature information includes: According to the formula , determine the state distance matrix D s , where s1 is the first state characteristic information, is the i-th historical state feature information, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
4. The method for real-time log analysis based on big data according to claim 2, characterized in that: Determining a center distance matrix according to the first state feature information, the historical state feature information and the target cluster center includes: According to the formula Determine the center distance matrix D o , where s1 is the first state characteristic information, is the i-th historical state feature information, o is the target cluster center, i is a positive integer, and i is less than or equal to m, and m is the number of historical feature information in the target cluster.
5. The method for real-time log analysis based on big data according to claim 2 is characterized in that: Processing the first state feature information, the historical state feature information, the state distance matrix, and the center distance matrix through the log file processing model to obtain the inferred feature information includes: According to the formula , obtain the output information OUT1 of the first processing level of the log file processing model, where D s is the state distance matrix, D o is the center distance matrix, I1 is the first state feature information, I h is the historical state feature information, L1 is the parameter matrix of the first processing level of the log file processing model; According to the formula , obtain the output information OUT of the jth processing level of the log file processing model j , where j is a positive integer greater than 1, L j is the parameter matrix of the jth processing level of the log file processing model; The inferred feature information is obtained according to output information of the last processing level of the log file processing model.
6. The method for real-time log analysis based on big data according to claim 5, characterized in that: When the inferred error is greater than or equal to a preset error threshold, the log file processing model is trained by using the inferred feature information and the target cluster center to obtain a trained log file processing model, including: According to the formula Get the loss function Loss; where o p is the inferred feature information, o is the target cluster center, The first state characteristic information I1 and the historical state characteristic information I h The column vector obtained by weighted averaging the column vectors of the matrix composed of The output information OUT of the kth processing level of the log file processing model k The column vector obtained by weighted average of multiple column vectors of α, β k , γ and μ are all preset weights, and the preset weights meet T is the number of historical cluster centers, o t is the t-th historical cluster center, t is a positive integer and t≤T; According to the loss function Loss, the log file processing model is trained to obtain a trained log file processing model.
7. The method for real-time log analysis based on big data according to claim 1, characterized in that: The method further comprises: Re-obtaining historical feature information of historical log files and first feature information of real-time log files through the trained log file processing model; An adjusted historical cluster center and an adjusted historical cluster cluster are obtained according to the re-acquired historical feature information, the re-acquired first feature information, the historical state information and the first operating state.
8. A real-time log analysis system based on big data, characterized in that: The system is arranged on a server and includes: The history module is used to obtain the historical feature information of the server's historical log files through the log file processing model; A clustering module, used for clustering the historical feature information and the historical status information of the server corresponding to the historical log file to obtain a plurality of historical clustering centers and historical clustering clusters, wherein the historical status information is the historical running status of the server at the time when the historical log file is generated; A real-time log module, configured to obtain, when a real-time log file is obtained, first characteristic information of the real-time log file and a first operating state of the server through the log file processing model; A target cluster module, used to determine the target cluster to which the real-time log file belongs according to the first characteristic information of the real-time log file, the first operating state and the multiple historical cluster centers; an inference module, configured to obtain inferred feature information of the real-time log file according to the first feature information, the first operating state, and the target cluster to which the real-time log file belongs, through the log file processing model; An error module, used to determine an inference error based on the inferred feature information and the target cluster center of the target cluster; A training module is used to train the log file processing model through the inferred feature information and the target cluster center to obtain a trained log file processing model when the inferred error is greater than or equal to a preset error threshold.
9. A real-time log analysis device based on big data, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.
10. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method according to any one of claims 1 to 7.
