Log data analysis method and device, equipment and storage medium

Abstract

The invention discloses a log data analysis method and device, equipment and a storage medium, and relates to the technical field of vehicle-mounted interconnection, and the method comprises the steps: obtaining log data, obtaining a key feature and an abnormal score according to the log data through a preset variational auto-encoder model, processing the log data according to the key feature and the abnormal score, and obtaining the log data. The processed log data are output to a visual window for image drawing analysis, and a display image is obtained. And log analysis, feature extraction, abnormity identification and visual display are performed by using the variational auto-encoder, so that the log analysis efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted interconnection technology, and in particular to a log data analysis method, device, equipment and storage medium. Background Art

[0002] As a frontier field of artificial intelligence and the automotive industry, log analysis plays a vital role in the software development process of intelligent driving technology. During the development, testing and operation of the autonomous driving system, a huge amount of log data will be generated. These logs record key information such as the system's operating status, sensor data, and decision-making process, and are an important basis for diagnosing system problems and optimizing algorithm performance. With the rapid development of intelligent driving technology, higher requirements are placed on the analysis and processing of log data, including processing speed, accuracy, degree of automation, and the ability to identify complex abnormal patterns.

[0003] At present, log analysis in the development of intelligent driving software is mainly based on engineers' experience, searching for specific keywords or error code markers in log files. This method relies on engineers' experience and understanding of the system, is inefficient, and has difficulty dealing with complex or unknown problems. Another method is to use the preset levels of the log system for preliminary screening. Although this method can quickly filter out some obvious problems, it has limited ability to conduct deeper analysis and identify abnormal patterns.

[0004] Although existing data analysis and display methods provide certain functions, they have the following defects in practical applications: they rely on predefined rules or simple statistical methods, and it is difficult to identify complex and potential abnormal patterns. Currently, log screening and analysis mainly rely on manual experience, which is time-consuming and inefficient in the face of massive data. Therefore, a more accurate and faster data analysis method is urgently needed to meet the growing testing and verification needs. Summary of the invention

[0005] The main purpose of this application is to provide a log data analysis method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the efficiency of log data analysis.

[0006] To achieve the above objectives, the present application proposes a log data analysis method, which includes:

[0007] Get log data;

[0008] Obtain key features and anomaly scores based on the log data through a preset variational autoencoder model;

[0009] Processing the log data according to the key features and the anomaly score to obtain processed log data;

[0010] The processed log data is output to a visualization window for image drawing and analysis to obtain a display image.

[0011] In one embodiment, the step of obtaining log data includes:

[0012] Get initial log data;

[0013] According to the initial log data, obtaining a timestamp and data information;

[0014] Perform elimination processing according to the data information to obtain processed log data;

[0015] The processed log data is converted into data of the same format according to the timestamp to obtain log data.

[0016] In one embodiment, before the step of obtaining key features and anomaly scores according to the log data by using a preset variational autoencoder model, the step includes:

[0017] Get log sample data and build an initial variational autoencoder;

[0018] The initial variational autoencoder is trained according to the log sample data to obtain a preset variational autoencoder model, wherein the initial variational autoencoder includes at least an encoding network layer, a latent space layer, and a decoding network layer, wherein the encoding network layer includes a first preset number of nodes, and the decoding network layer includes a second preset number of nodes.

[0019] In one embodiment, the step of training the initial variational autoencoder according to the log sample data to obtain a preset variational autoencoder model includes:

[0020] Initializing model parameters of the initial variational autoencoder;

[0021] Get log sample data;

[0022] Normalizing the log sample data to obtain processed input data;

[0023] Inputting the processed input data into the initial variational autoencoder to obtain predicted output data;

[0024] According to the loss function calculation, the error value between the predicted output data and the actual data is obtained;

[0025] Get the learning rate;

[0026] The model parameters are iteratively updated through an optimization algorithm according to the learning rate until a maximum number of iterations is reached or the error value calculated by the loss function converges to a preset threshold, thereby obtaining a preset variational autoencoder model.

[0027] In one embodiment, the step of obtaining key features and anomaly scores according to the log data by using a preset variational autoencoder model includes:

[0028] Inputting the log data into a preset variational autoencoder model to obtain key features;

[0029] Reconstructing the log data by using a preset variational autoencoder model to obtain reconstructed data;

[0030] Perform calculation based on the reconstructed data and the log data to obtain a reconstruction error;

[0031] An anomaly score is obtained based on the reconstruction error and the divergence calculation.

[0032] In one embodiment, the step of processing the log data according to the key features and the anomaly score to obtain processed log data includes:

[0033] Compare the abnormality score with a preset abnormality threshold to obtain a comparison result;

[0034] Classifying the key features based on the comparison results to obtain abnormal features;

[0035] Identify and filter corresponding abnormal log data according to the abnormal features to obtain processed log data.

[0036] In one embodiment, the step of outputting the processed log data to a visualization window for image drawing analysis to obtain a display image includes:

[0037] Obtaining the problem severity and log entries of the log data according to the processed log data;

[0038] The problem severity and the log entry are assigned corresponding visual attributes to the visualization window for drawing and analysis to obtain a display image, where the visual attributes at least include color, size, shape, and transparency.

[0039] In addition, to achieve the above purpose, the present application also proposes a log data analysis device, the log data analysis device comprising:

[0040] Acquisition module, used to obtain vehicle log data;

[0041] A calculation module, used for obtaining a sampling frequency according to the vehicle log data;

[0042] An analysis module, used to obtain analysis results according to the sampling frequency;

[0043] The drawing module is used to output the analysis result to a drawing window for image drawing to display the log data.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the log data analysis method described above are implemented.

[0045] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the log data analysis method described above are implemented.

[0046] This application obtains log data, obtains key features and anomaly scores based on the log data through a preset variational autoencoder model, processes the log data based on the key features and anomaly scores, obtains processed log data, and outputs the processed log data to a visualization window for image drawing analysis to obtain a display image. The variational autoencoder is used to analyze logs, extract features, identify anomalies, and perform visual display, thereby improving the efficiency of log analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 This is a flow chart of the first embodiment of the log data analysis method of the present application;

[0049] Figure 2 This is a variational autoencoder model structure diagram of the first embodiment of the log data analysis method of this application;

[0050] Figure 3 This is a log detection interface design diagram of the first embodiment of the log data analysis method of this application;

[0051] Figure 4 A visualized bubble chart of the first embodiment of the log data analysis method of the present application;

[0052] Figure 5 This is a flow chart of the second embodiment of the log data analysis method of the present application;

[0053] Figure 6 This is a schematic diagram of the module structure of the log data analysis device according to an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the device structure of the hardware operating environment involved in the log data analysis method in the embodiment of the present application;

[0055] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of the embodiment of the present application is: obtaining log data, obtaining key features and anomaly scores based on the log data through a preset variational autoencoder model, processing the log data according to the key features and anomaly scores to obtain processed log data, and outputting the processed log data to a visualization window for image drawing and analysis to obtain a display image.

[0059] Based on this, the present application embodiment provides a log data analysis method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the log data analysis method of the present application.

[0060] In this embodiment, the log data analysis method includes steps S10 to S40:

[0061] Step S10, obtaining log data.

[0062] It should be noted that log data usually includes the vehicle's sensor readings, controller status, actuator feedback, and environmental information, which are transmitted in real time through vehicle communication networks such as CAN or Ethernet. The collection of log data not only involves real-time recording of data, but also includes data preprocessing, storage, and subsequent analysis. In order to ensure the integrity and accuracy of the data, it is usually necessary to verify and synchronize the data to ensure time consistency between different data sources.

[0063] As the amount of data increases, traditional data processing methods cannot meet the needs of efficient analysis. This embodiment can receive and process large-scale vehicle log data based on the log analysis and drawing tool of this embodiment. The tool first preprocesses the log data, including data cleaning, format unification and outlier detection.

[0064] Further, the initial log data is obtained, and the timestamp and data information are obtained according to the initial log data. The data information is eliminated to obtain the processed log data, and the processed log data is converted into data of the same format according to the timestamp to obtain the log data. The initial log data obtained from the vehicle contains information from different sources, such as vehicle status data collected through the CAN bus or text logs stored in CSV or TXT format. These raw data usually have their own timestamps and specific data formats. In order to enable these diverse data to be analyzed and processed uniformly, the timestamps and data information are first extracted, and then the data information is preprocessed, including eliminating irrelevant or redundant information to ensure that valid data is retained, which can improve the efficiency and accuracy of subsequent processing. Subsequently, the processed log data is normalized and converted according to the unified timestamp format so that all log data have the same format, which is convenient for further time series analysis and pattern recognition. Through this series of operations, a structured and uniformly formatted log data set is finally obtained.

[0065] Furthermore, for log data, a counting-based unified vectorization method is adopted. The specific steps are as follows: First, define an n-dimensional vector V, where n is equal to the size of the character set Σ, which contains all uppercase letters AZ, lowercase letters az, numbers 0-9, and common punctuation marks. This means that each dimension of the vector V corresponds to a character in the character set. Secondly, define a mapping function f, which maps each character in the character set Σ to an integer, that is, f:Σ→{1,2,...,n}. For example, the character 'A' is mapped to 1, the character 'B' is mapped to 2, and so on. For any given log sentence S, we first convert it into a vector representation v(S), the length of which is n. Each component e[i] of the vector v(S) represents the number of occurrences of the i-th character in the character set Σ in the sentence S. Specifically, if a character c appears multiple times in the sentence S, then the component value corresponding to the character in the vector v(S) is the number of times it appears in S. For example, if the character 'A' appears 3 times in sentence S, the position corresponding to 'A' in vector v(S) will have a value of 3; if a character does not appear in sentence S, the value of the corresponding position of the character is 0. In this way, the original log text is converted into a numerical vector, which can better capture the potential features in the log data and perform anomaly detection through reconstruction error and KL divergence.

[0066] Step S20, obtaining key features and anomaly scores based on the log data through a preset variational autoencoder model.

[0067] It should be noted that the log data is processed by the preset variational autoencoder model. As a deep learning model, the variational autoencoder can effectively map high-dimensional log data into a low-dimensional latent space. In this process, it can not only capture the main features of the data, but also evaluate the abnormality of the data through reconstruction error. First, the preprocessed structured log data is input into the encoder, which maps the data into a low-dimensional latent space and extracts the key features of the log data. During the training process, the variational autoencoder optimizes the model parameters by minimizing the reconstruction error and KL divergence. The reconstruction error measures the difference between the input data and its reconstructed version, while the KL divergence measures the difference between the posterior distribution generated by the encoder and the preset prior distribution. The reconstruction error and KL divergence are combined to obtain the abnormality score.

[0068] Furthermore, before step S20, it also includes obtaining log sample data, establishing an initial variational autoencoder, training the initial variational autoencoder according to the log sample data, and obtaining a preset variational autoencoder model, such as Figure 2The variational autoencoder model structure diagram shown in the figure, wherein the initial variational autoencoder includes an encoding network layer, a latent space layer and a decoding network layer, the encoding network layer includes a first preset number of nodes, and the decoding network layer includes a second preset number of nodes. Before using the preset variational autoencoder model, it is first necessary to select a portion of log sample data as a training set, and these sample data should be able to fully represent the log features generated by the system under normal operating conditions. Then, an initial variational autoencoder model is constructed, which consists of an encoding network layer, a latent space layer and a decoding network layer. The encoding network layer is responsible for compressing high-dimensional log data into a low-dimensional latent space representation, and usually contains a first preset number of nodes to adapt to the dimension and complexity of the input data; the decoding network layer is used to reconstruct data from the latent space, and it contains a second preset number of nodes, aiming to restore log data as close to the original input as possible. By adjusting the weight parameters in the encoder and decoder, the model continuously learns how to efficiently encode and decode log data during the training process. For the specific steps of the model, first initialize the model parameters of the initial variational autoencoder, including the weights and bias items in the encoding network layer and the decoding network layer, and these parameters are usually randomly set to values ​​within a small range. Next, obtain a batch of representative log sample data, which reflects the typical situation when the system is operating normally. In order to ensure the stability and efficiency of model training, these log sample data need to be normalized to standardize their numerical range and obtain processed input data. Subsequently, the processed input data is input into the initial variational autoencoder, and the model generates predicted output data through forward propagation. The error value between the predicted output data and the original log sample data is calculated using the loss function. This step evaluates the performance of the model in the current state. Obtain an appropriate learning rate, which determines the magnitude of parameter adjustment each time the gradient decreases. Using optimization algorithms such as stochastic gradient descent (SGD), Adam, etc., the model parameters are iteratively updated according to the learning rate. This process will continue until the preset maximum number of iterations is reached or the error value calculated by the loss function converges to a preset threshold. When the stopping condition is met, the final result is a preset variational autoencoder model that has been trained and optimized.

[0069] Step S30: Process the log data according to the key features and the anomaly scores to obtain processed log data.

[0070] It should be noted that in the process of processing log data according to key features and anomaly scores, each log data is first encoded through the preset variational autoencoder model to extract key features that reflect its intrinsic characteristics. These features may be the changing trend of system performance indicators, the frequency of occurrence of specific events, or other information that helps to understand the system status. At the same time, the model will also assign an anomaly score to each log data, which quantifies the degree of deviation between the log data and the known normal behavior pattern. The higher the score, the more likely the entry is to represent an abnormal condition in the system. Based on these scores, certain thresholds can be set to distinguish normal logs from suspected abnormal logs. Logs below the threshold are considered normal and can be filtered out or archived, while logs above the threshold are marked as potential problem points that require further analysis or immediate attention.

[0071] Further, step S30 includes: comparing the anomaly score with a preset anomaly threshold to obtain a comparison result. Comparing the anomaly score with the preset anomaly threshold, first calculate the anomaly score of each log data after passing through the variational autoencoder. The score combines the results of the reconstruction error and the KL divergence, reflecting the degree of deviation of the log data from the normal behavior pattern. Subsequently, the calculated anomaly score is compared with the preset anomaly threshold. The preset anomaly threshold is an empirical value determined based on historical data or through cross-validation to distinguish normal behavior from potential anomalies, and then a comparison result is obtained. If the anomaly score of a log exceeds this threshold, it indicates that the log data may contain signs of system anomalies and requires further investigation; conversely, if the score is lower than the threshold, it is considered that the log is within the normal range and can be filtered out or archived.

[0072] Furthermore, step S30 also includes: classifying key features based on the comparison results to obtain abnormal features. When the log data anomaly score calculated by the variational autoencoder model exceeds the preset anomaly threshold, it means that the log entry contains information about system anomalies. At this point, the system will further analyze the key features that lead to high anomaly scores, including but not limited to the frequency of occurrence of specific events, abnormal values ​​of sensor data, unexpected branches in decision logic, etc. By classifying these features, the system can identify which specific patterns or combinations of variables constitute abnormal features. The identification of abnormal features helps developers quickly locate the root cause of the problem because they provide clues about potential fault points in the system. For example, if the abnormal features of a log appear concentrated on the sensor readings of the vehicle perception module, this may indicate a problem with the sensor hardware or software.

[0073] Furthermore, step S30 also includes: identifying and filtering corresponding abnormal log data according to abnormal features to obtain processed log data. When the log data anomaly score calculated by the variational autoencoder model exceeds the preset anomaly threshold, it indicates that these log data may contain signs of system anomalies. The system will perform a detailed feature analysis on these high-scoring log data to identify key features that lead to increased anomaly scores, such as abnormal fluctuations in sensor readings, unexpected behaviors in decision logic, or any other system state changes that do not conform to normal patterns. Once these abnormal features are confirmed, the system will mark the log entries containing these features as abnormal log data. Next, the system will filter these abnormal log data, separate them from regular log data and save them for subsequent in-depth analysis. In addition to filtering the log data according to the variational autoencoder model, you can also use Figure 3 The log detection interface obtained from the log detection interface design diagram shown cleans and vectorizes the log files.

[0074] Step S40, outputting the processed log data to a visualization window for image drawing and analysis to obtain a display image.

[0075] It should be noted that according to the processed log data output to the visualization window for image drawing analysis, the log data processed and filtered by the variational autoencoder is first imported into the visualization component. These data have been classified and a large number of normal log entries have been removed, leaving only log entries with high abnormality scores, which contain clues to potential problems in the system. In the visualization window, the system uses customized graphical tools such as Figure 4 The log data is presented in a visual bubble chart as shown. Each bubble represents a log entry or a category of log entries after analysis and filtering. The color and size of the bubble can be set according to the severity and frequency of the log problem. The darker the color and the larger the bubble, the more serious the problem or the higher the frequency of occurrence. In addition, through the design of the timeline, users can clearly see the changing trend of these abnormal events over time, helping to identify whether there are regular problems or sudden changes in system performance within a specific time period. When the user hovers the mouse over any bubble, detailed information about the log entry is displayed, including timestamp, exception type, associated system modules, etc. Such an interactive design allows developers to quickly locate the exact time and location of the problem, as well as the possible scope of impact.

[0076] Further, step S40 is more specifically described as obtaining the problem severity and log entries of the log data according to the processed log data, assigning the corresponding visual attributes to the visualization window for drawing and analysis, and obtaining a display image, wherein the visual attributes at least include color, size, shape, and transparency. According to the processed log data, the problem severity of each log is determined by the log data anomaly score calculated by the variational autoencoder, and specific visual attributes such as color, size, shape, and transparency are assigned to each log entry accordingly, so as to facilitate drawing and analysis in the visualization window. Color can be used to distinguish different levels of problem severity, such as red for serious errors, yellow for warnings, and green for normal log entries; size can reflect the frequency of occurrence of problems or the scope of impact, the more frequent the problem or the greater the impact, the larger the corresponding graphic element; the choice of shape can be determined according to the type of log entry or the system module to which it belongs, such as a circle for logs from a sensor, and a square for logs from a decision module; transparency can be adjusted according to the newness of the log data, the latest log entry has a higher transparency, and the older log has a lower transparency.

[0077] This embodiment provides a log data analysis method, which obtains log data, obtains key features and anomaly scores based on the log data through a preset variational autoencoder model, processes the log data based on the key features and anomaly scores, obtains processed log data, and outputs the processed log data to a visualization window for image drawing analysis to obtain a display image. The variational autoencoder is used to analyze logs, extract features, identify anomalies, and perform visual display, thereby improving the efficiency of log analysis.

[0078] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 5 The log data analysis method step S20 also includes steps S201 to S204:

[0079] Step S201, input the log data into a preset variational autoencoder model to obtain key features.

[0080] It should be noted that the original log data is first mapped into a low-dimensional latent space through the encoder part of the model, which involves extracting key features from the data. The encoder reduces the dimensionality of the input data through the neural network layer to capture the key features in the log data.

[0081] Step S202, reconstructing the log data through a preset variational autoencoder model to obtain reconstructed data.

[0082] It should be noted that the preprocessed log data is input into the encoder part of the model, which compresses the log data into a low-dimensional latent representation that captures the key features of the original log data. Subsequently, the latent representation is fed into the decoder, whose task is to regenerate log data similar to the original input based on the representation in the latent space. This reconstruction process utilizes the power of the variational autoencoder, and by minimizing the difference between the original log data and the reconstructed data, the model learns how to efficiently represent and reconstruct the log data. The difference between the reconstructed data and the original data can be used to evaluate the performance of the model.

[0083] Step S203, performing calculations based on the reconstructed data and the log data to obtain a reconstruction error.

[0084] It should be noted that the original log data is encoded into a latent representation through the variational autoencoder, and then the latent representation is reconstructed back into the form of the original data through the decoder to obtain the reconstructed data. Next, the difference between the reconstructed data and the original log data is calculated. This difference is called the reconstruction error. The reconstruction error can be calculated using a variety of metrics, such as mean square error (MSE), cross entropy, etc., depending on the specific nature of the log data and the goal of the model. This error reflects the accuracy of the model in reconstructing the log data. A large error means that the model fails to capture the characteristics of the original data well in some aspects, which is due to abnormal patterns in the data.

[0085] Step S204, obtaining an abnormality score according to the reconstruction error and the divergence calculation.

[0086] It should be understood that the input log data sentence vector is defined as X, and the reconstruction vector output by the encoder part of the variational autoencoder is X'. The calculation formula of the reconstruction error can be expressed as:

[0087] R=||X i -X' i || 2

[0088] The reconstruction error reflects the difference between the original data and the reconstructed data. The smaller the difference, the stronger the model's ability to fit the data.

[0089] Next, we calculate the KL divergence, which is a way to measure the difference between two probability distributions. In the variational autoencoder, it is used to make the prior distribution p(z) in the latent space close to the posterior distribution q(z|x) simulated by the encoder. The formula for KL divergence is:

[0090] D KL =(q(z|X i )|p(z))

[0091] It quantifies the distance between the posterior distribution generated by the encoder and the preset prior distribution. Ideally, this value should be as small as possible, indicating that the distribution generated by the encoder is close to the preset simple distribution.

[0092] Finally, when calculating the anomaly score, we need to combine the reconstruction error and KL divergence. The anomaly score can be expressed as:

[0093] A=α*R+(1-α)*D KL

[0094] Where α is a balancing factor that adjusts the relative importance between reconstruction error and KL divergence. This balancing factor can be determined by cross-validation to ensure that the model performs well on the training set and has good generalization ability on unknown data. In this way, the anomaly score combines the reconstruction error of the data and the difference between the encoder-generated distribution and the prior distribution, providing a comprehensive indicator to evaluate the degree of abnormality of the log data. Log data with high anomaly scores indicate that these data deviate greatly from the normal patterns learned by the model, indicating that there are abnormalities in the system.

[0095] This embodiment obtains key features by inputting log data into a preset variational autoencoder model, reconstructs the log data through the preset variational autoencoder model to obtain reconstructed data, performs calculations based on the reconstructed data and the log data to obtain a reconstruction error, and obtains anomaly scores based on the reconstruction error and divergence calculation, thereby effectively identifying anomalies and improving the accuracy of system anomaly analysis.

[0096] This application also provides a log data analysis device, please refer to Figure 6 , the device comprises:

[0097] The acquisition module 10 is used to acquire log data.

[0098] The processing module 20 is used to obtain key features and anomaly scores according to the log data through a preset variational autoencoder model.

[0099] The filtering module 30 is used to process the log data according to the key features and the anomaly scores to obtain processed log data.

[0100] The display module 40 is used to output the processed log data to the visualization window for image drawing and analysis to obtain a display image.

[0101] The log data analysis device provided by the present application adopts the log data analysis method in the above embodiment, which can solve the technical problem of how to improve the efficiency of log data analysis. Compared with the prior art, the beneficial effects of the log data analysis device provided by the present application are the same as the beneficial effects of the log data analysis method provided by the above embodiment, and the other technical features in the log data analysis device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0102] In one embodiment, the acquisition module 10 is also used to acquire initial log data; obtain a timestamp and data information based on the initial log data; perform elimination processing based on the data information to obtain processed log data; convert the processed log data into data of the same format based on the timestamp to obtain log data.

[0103] In one embodiment, the processing module 20 is also used to obtain log sample data and establish an initial variational autoencoder; the initial variational autoencoder is trained according to the log sample data to obtain a preset variational autoencoder model, and the initial variational autoencoder includes at least an encoding network layer, a latent space layer and a decoding network layer, the encoding network layer includes a first preset number of nodes, and the decoding network layer includes a second preset number of nodes.

[0104] In one embodiment, the processing module 20 is also used to initialize the model parameters of the initial variational autoencoder; obtain log sample data; normalize the log sample data to obtain processed input data; input the processed input data into the initial variational autoencoder to obtain predicted output data; calculate according to the loss function to obtain the error value between the predicted output data and the actual data; obtain the learning rate; iteratively update the model parameters through the optimization algorithm according to the learning rate until the maximum number of iterations is reached or the error value calculated by the loss function converges to a preset threshold, thereby obtaining a preset variational autoencoder model.

[0105] In one embodiment, the processing module 20 is also used to input the log data into a preset variational autoencoder model to obtain key features; reconstruct the log data through the preset variational autoencoder model to obtain reconstructed data; calculate based on the reconstructed data and the log data to obtain a reconstruction error; and calculate based on the reconstruction error and divergence to obtain an anomaly score.

[0106] In one embodiment, the filtering module 30 is further used to compare the abnormal score with the preset abnormal threshold to obtain a comparison result; classify the key features based on the comparison result to obtain abnormal features; identify and filter the corresponding abnormal log data according to the abnormal features to obtain processed log data.

[0107] In one embodiment, the display module 40 is also used to obtain the problem severity and log entries of the log data based on the processed log data; assign corresponding visual attributes of the problem severity and the log entries to the visualization window for drawing and analysis to obtain a display image, and the visual attributes include at least color, size, shape, and transparency.

[0108] The present application provides a log data analysis device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the log data analysis method in the above-mentioned embodiment one.

[0109] Reference below Figure 7 , which shows a schematic diagram of the structure of a log data analysis device suitable for implementing the embodiment of the present application. The log data analysis device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The log data analysis device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0110] like Figure 7As shown, the log data analysis device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the log data analysis device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the log data analysis device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a log data analysis device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0111] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0112] The log data analysis device provided by the present application adopts the log data analysis method in the above embodiment, which can solve the technical problem of how to improve the efficiency of log data analysis. Compared with the prior art, the beneficial effects of the log data analysis device provided by the present application are the same as the beneficial effects of the log data analysis method provided by the above embodiment, and the other technical features in the log data analysis device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0113] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0114] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0115] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the log data analysis method in the above-mentioned embodiment.

[0116] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible storage medium that contains or stores a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate storage medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0117] The computer-readable storage medium may be included in the log data analysis device; or may exist independently without being assembled into the log data analysis device.

[0118] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the log data analysis device, the log data analysis device can write computer program codes for performing the operations of the present application in one or more programming languages ​​or a combination thereof. The programming languages ​​include object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0121] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned log data analysis method, and can solve the technical problem of how to improve the efficiency of log data analysis. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the log data analysis method provided by the above-mentioned embodiment, and will not be repeated here.

[0122] The present application also provides a computer program product, including a computer program, which implements the steps of the log data analysis method as described above when executed by a processor.

[0123] The computer program product provided by this application can solve the technical problem of how to improve the efficiency of log data analysis. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the log data analysis method provided by the above embodiment, which will not be repeated here.

[0124] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A log data analysis method, characterized in that: The method comprises: Get log data; Obtain key features and anomaly scores based on the log data through a preset variational autoencoder model; Processing the log data according to the key features and the anomaly score to obtain processed log data; The processed log data is output to a visualization window for image drawing and analysis to obtain a display image.

2. The method according to claim 1, characterized in that The step of obtaining log data includes: Get initial log data; According to the initial log data, obtaining a timestamp and data information; Perform elimination processing according to the data information to obtain processed log data; The processed log data is converted into data of the same format according to the timestamp to obtain log data.

3. The method according to claim 1, characterized in that Before the step of obtaining key features and abnormality scores according to the log data by using a preset variational autoencoder model, the method includes: Get log sample data and build an initial variational autoencoder; The initial variational autoencoder is trained according to the log sample data to obtain a preset variational autoencoder model, wherein the initial variational autoencoder includes at least an encoding network layer, a latent space layer, and a decoding network layer, wherein the encoding network layer includes a first preset number of nodes, and the decoding network layer includes a second preset number of nodes.

4. The method according to claim 3, characterized in that The step of training the initial variational autoencoder according to the log sample data to obtain a preset variational autoencoder model includes: Initializing model parameters of the initial variational autoencoder; Get log sample data; Normalizing the log sample data to obtain processed input data; Inputting the processed input data into the initial variational autoencoder to obtain predicted output data; According to the loss function calculation, the error value between the predicted output data and the actual data is obtained; Get the learning rate; The model parameters are iteratively updated through an optimization algorithm according to the learning rate until a maximum number of iterations is reached or the error value calculated by the loss function converges to a preset threshold, thereby obtaining a preset variational autoencoder model.

5. The method according to claim 1, characterized in that The step of obtaining key features and anomaly scores according to the log data by using a preset variational autoencoder model includes: Inputting the log data into a preset variational autoencoder model to obtain key features; Reconstructing the log data by using a preset variational autoencoder model to obtain reconstructed data; Perform calculation based on the reconstructed data and the log data to obtain a reconstruction error; An anomaly score is obtained based on the reconstruction error and the divergence calculation.

6. The method according to claim 1, characterized in that The step of processing the log data according to the key features and the anomaly score to obtain processed log data includes: Compare the abnormality score with a preset abnormality threshold to obtain a comparison result; Classifying the key features based on the comparison results to obtain abnormal features; Identify and filter corresponding abnormal log data according to the abnormal features to obtain processed log data.

7. The method according to claim 1, characterized in that The step of outputting the processed log data to a visualization window for image drawing and analysis to obtain a display image comprises: Obtaining the problem severity and log entries of the log data according to the processed log data; The problem severity and the log entry are assigned corresponding visual attributes to the visualization window for drawing to obtain a display image, where the visual attributes at least include color, size, shape, and transparency.

8. A log data analysis device, characterized in that: The device comprises: Acquisition module, used to obtain log data; A processing module, used to obtain key features and anomaly scores according to the log data through a preset variational autoencoder model; A filtering module, used to process the log data according to the key features and the anomaly score to obtain processed log data; The display module is used to output the processed log data to a visualization window for image drawing and analysis to obtain a display image.

9. A log data analysis device, characterized in that: The device comprises: a memory, a processor, and a log data analysis program stored in the memory and executable on the processor, wherein the log data analysis program is configured to implement the steps of the log data analysis method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a log data analysis program, which, when executed by a processor, implements the steps of the log data analysis method according to any one of claims 1 to 7.

Images