Log data processing method and device based on CMDB

Abstract

The invention discloses a log data processing method and device based on CMDB. The method comprises the following steps: classifying to-be-screened log data by using a trained GRU model, determining the complexity or simplicity of the to-be-screened log data, and performing structured processing on the complex log data in combination with a CMDB configuration item; the GRU model training process comprises the steps that historical log data are classified and marked to obtain a log sample data set, the log sample data set comprises the historical log data and corresponding classification marks, and the classification marks comprise simple marks and complex marks; and inputting historical log data in the log sample data set into the constructed GRU model for learning training, determining model loss and adjusting model parameters according to a classification result output by the model and a classification mark in the log sample data set, and obtaining a trained GRU model of which the model loss meets requirements through repeated iterative training. Complicated log data can be screened out for structuralization before the log data is structuralized, waste of time and resources is avoided, and subsequent log audit and query are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of log processing, and in particular to a log data processing method and device based on a configuration management database (CMDB). Background Art

[0002] Logs, as a record of crucial information during system operation, are widely used by service personnel and technicians to determine system availability, correctness, and stability. Existing technologies for automated operations and maintenance centralize log management, previously distributed across various machines. Regular expressions are defined to structure logs and extract key information from them for monitoring and alerting.

[0003] A Chinese patent with announcement number CN111930701B mentions a log structured processing method and device. This method implements local sequence comparison calculation of logs based on the Smith-Waterman distance algorithm, performs log clustering analysis through the Drains algorithm, and realizes rapid streaming processing of log structured data. Summary of the Invention

[0004] The inventors of the present application have discovered that existing log structuring processing solutions based on regular expressions or other algorithms do not fully screen the log data before structuring it. Both simple and complex log data are uniformly structured, and structuring of simple log data is unnecessary, which will result in extended processing time and waste of resources, and is not conducive to subsequent log auditing and query work.

[0005] In view of the above problems, the present invention is proposed to provide a CMDB-based log data processing method, device and system that overcome the above problems or at least partially solve the above problems.

[0006] An embodiment of the present invention provides a log data processing method based on CMDB, comprising:

[0007] Use the trained gated recurrent unit (GRU) model to classify the log data to be filtered, and determine whether the log data to be filtered is complex log data or simple log data; and perform structured processing on the complex log data in combination with CMDB configuration items;

[0008] The process of obtaining the trained GRU model includes:

[0009] Classifying and labeling the acquired historical log data to obtain a log sample data set, wherein the log sample data set includes the historical log data and corresponding classification labels, wherein the classification labels include simple and complex;

[0010] Input the historical log data in the log sample dataset into the constructed GRU model for learning and training. Determine the model loss and adjust the model parameters according to the classification results output by the model and the classification labels in the log sample dataset. After multiple iterative trainings, obtain a trained GRU model with a model loss meeting the requirements.

[0011] In some alternative embodiments, the classifying and labeling the obtained historical log data to obtain a log sample dataset includes:

[0012] Obtain historical log data from an operating system or an application;

[0013] Classify the historical log data according to the set log classification rules, determine whether the historical log data belongs to complex log data or simple log data, and add a classification identifier to the historical log data according to the classification result; the log classification rules are set according to the log volume, the number of examples, and the complexity of the Configuration Management Database (CMDB).

[0014] Convert the historical log data into log vector data through a word embedding text feature extraction method, and construct a log sample dataset including the log vector data and the corresponding classification labels.

[0015] In some alternative embodiments, the converting the historical log data into log vector data through a word embedding text feature extraction method includes:

[0016] For each historical log data, map the word sequence in the historical log data to the corresponding word embedding vector;

[0017] Perform an average or weighted average on the word embedding vectors of each word sequence to obtain a vector representation of the historical log data;

[0018] The vector representation of the historical log data matches the input format of the input layer of the GRU model.

[0019] In some alternative embodiments, the GRU model includes an input layer, a GRU layer, and an output layer;

[0020] The input layer is used for inputting the log vector data, the GRU layer is used for processing the log vector data and extracting the feature data of the log vector data, and the output layer is used for outputting the log classification result based on the feature data of the log vector data.

[0021] In some alternative embodiments, it further includes:

[0022] Integrate the trained GRU model and the CMDB system into the log processing system; the GRU model is used to obtain the log data to be screened or historical log data from the log processing system, and feedback the classified result of the log data to the log processing system; the CMDB system is used to store CMDB configuration items.

[0023] In some optional embodiments, the structured processing of complex log data in combination with CMDB configuration items includes:

[0024] According to the unique identifier and configuration information included in the CMDB configuration item, perform association and clustering analysis on the determined complex log data, and perform structured processing on the complex log data according to the results of the association and clustering analysis.

[0025] In some optional embodiments, according to the unique identifier and configuration information included in the CMDB configuration item, performing association and clustering analysis on the determined complex log data includes:

[0026] According to the unique identifier included in the CMDB configuration item, determine whether the devices or components involved in the complex log data are the devices or components recorded in the CMDB system;

[0027] If not, send an unregistered reminder message; if so, according to the configuration item corresponding to the unique identifier, determine the associated devices, components and historical log data; perform clustering on the complex log data according to the associated devices, components and historical log data; based on the clustering result, obtain the structured data of the complex log data.

[0028] In some optional embodiments, it further includes:

[0029] Update the CMDB configuration item according to the identifier and configuration information included in the determined complex log data.

[0030] An embodiment of the present invention further provides a log data processing device, including:

[0031] A log classification module, configured to classify the log data to be screened using a trained gated recurrent unit GRU model, and determine that the log data to be screened is complex log data or simple log data;

[0032] A data processing module, configured to classify and label the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex;

[0033] A model training module, configured to input the historical log data in the log sample dataset into a constructed GRU model for learning and training, determine the model loss and adjust the model parameters according to the classification result output by the model and the classification labels in the log sample dataset, and after multiple iterative trainings, obtain a trained GRU model with a model loss meeting the requirements.

[0034] In some optional embodiments, the log data processing device further includes:

[0035] A structuring processing module, configured to perform association and clustering analysis on the determined complex log data according to the unique identifier and configuration information included in the CMDB configuration item, and perform structuring processing on the complex log data according to the association and clustering analysis results.

[0036] An embodiment of the present invention further provides a log data processing system, in which a CMDB system and a GRU model are integrated;

[0037] The GRU model is configured to classify the log data to be screened generated in the log data processing system to determine whether the log data to be screened is complex log data or simple log data; the process of obtaining the trained GRU model includes: classifying and labeling the acquired historical log data to obtain a log sample dataset, where the log sample dataset includes historical log data and corresponding classification labels, and the classification labels include simple and complex; inputting the historical log data in the log sample dataset into a constructed GRU model for learning and training, determining the model loss and adjusting the model parameters according to the classification result output by the model and the classification labels in the log sample dataset, and after multiple iterative trainings, obtaining a trained GRU model with a model loss meeting the requirements;

[0038] The CMDB system is configured to store CMDB configuration items;

[0039] The log processing system is configured to perform association and clustering analysis on the determined complex log data according to the classification result of the GRU model and the CMDB configuration item, and perform structuring processing on the complex log data according to the association and clustering analysis results.

[0040] An embodiment of the present invention further provides a computer storage medium, characterized in that the computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the above-mentioned CMDB-based log data processing method is implemented.

[0041] An embodiment of the present invention further provides a log data processing device, which is characterized by including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the log data processing method based on CMDB as described above.

[0042] The beneficial effects of the above technical solutions provided by the embodiments of the present invention at least include:

[0043] The log data processing method based on CMDB provided by the embodiments of the present invention classifies and marks the obtained historical log data simply or complexly to obtain a log sample data set, inputs the historical log data in the log sample data set into the constructed GRU model for learning and training, determines the model loss and adjusts the model parameters according to the classification results output by the model and the classification marks in the log sample data set, and after multiple iterative trainings, obtains a trained GRU model with a model loss meeting the requirements; classifies the log data to be screened based on the trained GRU model to determine whether the log data to be screened is complex log data or simple log data. Through this method, complex log data can be screened out for structuring before structuring the log data, avoiding structuring simple data, avoiding waste of time and resources, and being more conducive to subsequent log auditing and query work.

[0044] In the above method, the GRU model and the CMDB system can also be integrated into the log processing system together, so as to realize classifying logs using the trained GRU model, and performing clustering analysis and processing on complex log data in combination with CMDB configuration items, mining the association relationships between logs, and whether the devices and / or components involved in the logs are recorded devices and components, associating and clustering the logs from the same device or component, and better mining the useful information in the logs, so as to obtain better structured processing results.

[0045] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings.

[0046] The technical solutions of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0047] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0048] Figure 1Flowchart of the log data processing method based on CMDB in an embodiment of the present invention;

[0049] Figure 2 Structural schematic diagram of the log data processing device based on CMDB in an embodiment of the present invention;

[0050] Figure 3 Structural schematic diagram of the log data processing system based on CMDB in an embodiment of the present invention. Detailed implementation manners

[0051] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0052] To solve the problem in the prior art that there is no sufficient screening before structuring log data, and for simple data, structuring will instead waste time and resources, which is not conducive to subsequent log auditing and query work, an embodiment of the present invention provides a log data processing method based on CMDB, and the method includes:

[0053] Using a trained gated recurrent unit (GRU) model to classify the log data to be screened, and determining whether the log data to be screened is complex log data or simple log data; Structuring the complex log data in combination with CMDB configuration items; wherein, the process of obtaining the trained GRU model includes:

[0054] Classifying and labeling the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex;

[0055] Inputting the historical log data in the log sample data set into the constructed GRU model for learning and training, determining the model loss and adjusting the model parameters according to the classification result output by the model and the classification labels in the log sample data set, and after multiple iterative trainings, obtaining a trained GRU model with a model loss meeting the requirements.

[0056] The above-mentioned CMDB-based log data processing method classifies and marks the obtained historical log data, whether simple or complex, to obtain a log sample data set. The historical log data in the log sample data set is input into the constructed GRU model for learning and training. According to the classification results output by the model and the classification marks in the log sample data set, the model loss is determined and the model parameters are adjusted. After multiple iterative trainings, a trained GRU model with a model loss meeting the requirements is obtained. Based on the trained GRU model, complex log data is screened out, and the complex log data is structurally processed in combination with CMDB configuration items. Through this method, before structuring the log data, complex log data can be screened out for structuring, avoiding structuring simple data, thus avoiding waste of time and resources, and being more conducive to subsequent log auditing and query work.

[0057] In the above method, the GRU model and the CMDB system can also be integrated into the log processing system together, so as to realize classifying logs using the trained GRU model, and performing clustering analysis and processing on complex log data in combination with CMDB configuration items, mining the correlation relationships between logs, and determining whether the devices and / or components involved in the logs are recorded devices and components. For logs from the same device or component, they are associated and clustered to better mine useful information in the logs, thereby obtaining better structured processing results.

[0058] In the above method, the trained GRU model is used to realize the classification of log data, and clustering analysis of the log data is performed according to the classification results to better realize structured processing. A specific implementation process of this method is as Figure 1 shown, including:

[0059] S101: Obtain historical log data, classify and mark the obtained historical log data to obtain a log sample data set. The log sample data set includes historical log data and corresponding classification marks, and the classification marks include simple and complex;

[0060] In this step, the data format of the historical log data can be determined according to subsequent needs. For example, it can be log vector data.

[0061] The process of obtaining the log sample data set includes: obtaining historical log data from the operating system or application program; classifying the historical log data according to the set log classification rules, determining whether the historical log data belongs to complex log data or simple log data, and adding a classification identifier to the historical log data according to the classification results; converting the historical log data into log vector data through a word embedding text feature extraction method, and constructing a log sample data set including the log vector data and corresponding classification marks. Among them, the log classification rules are set according to the log volume, example volume, and the complexity of the management database CMDB.

[0062] When obtaining historical log data from an operating system or an application, the historical log data can be directly obtained from the operating system or the application, or a log aggregation tool can be used for obtaining; the obtained historical log data can include simple log data and complex log data; after obtaining the historical log data, combined with different scenarios, based on log classification rules, the historical log data is classified and marked. The log aggregation tool used can be selected as needed, such as log aggregation tools like ELK Stack, Splunk, Vector, etc. The marked log data can be stored in a distributed file system, and according to the needs of model training, using the word embedding text feature extraction method, the log text is converted into a vector, that is, converted into a digital representation acceptable to a recurrent neural network (RNN).

[0063] Converting the historical log data into log vector data through the word embedding text feature extraction method includes: for each historical log data, mapping the word sequence in the historical log data to the corresponding word embedding vector; performing an average or weighted average on the word embedding vectors of each word sequence to obtain the vector representation of the historical log data; wherein, the vector representation of the historical log data matches the input format of the input layer of the GRU model.

[0064] Optionally, the specific process of implementing word embedding includes:

[0065] 1) Based on the collected historical log data, collect all different words in the log data, and assign a unique integer identifier to each word to make a vocabulary.

[0066] 2) For each historical log data, perform word segmentation on the historical log data, query the corresponding integer identifier of each word based on the vocabulary, and obtain the integer identifier sequence of the historical log data based on the corresponding integer identifier of each word, so as to realize the conversion of the word-segmented historical log data into an integer identifier sequence.

[0067] 3) Perform word embedding training based on the integer identifier sequence corresponding to the historical log data, convert the integer identifier sequence into batch data to be used in word embedding training, that is, training samples, to train the word embedding model, and use the trained word embedding model to convert each word into the corresponding word embedding vector.

[0068] 4) According to the word embedding vector corresponding to each word, obtain the vector representation of the historical log data.

[0069] Through the above steps, the word embedding vector corresponding to each word in the historical log data is obtained, and an average or weighted average is performed on the word embedding vectors of each word to obtain the vector representation of the historical log data, and each historical log data is represented as a vector with a fixed length.

[0070] S102: Construct a GRU model.

[0071] The constructed GRU model includes an input layer, a GRU layer, and an output layer; the input layer is used for inputting log vector data, the GRU layer is used for processing log vector data and extracting feature data of the log vector data, and the output layer is used for outputting a log classification result based on the feature data of the log vector data. The GRU model has better long-term dependence modeling ability, more comprehensive context modeling ability, and stronger long-term memory ability, requires less training time, and has higher training data efficiency.

[0072] In this embodiment, the input of the input layer of the GRU model is the vector representation of the log text, the GRU layer is used for processing sequence data, and the output of the output layer is a binary classification result, which can be represented by 0 or 1, corresponding to simple or complex in sequence, to represent simple log data or complex log data.

[0073] The above steps S101 and S102 are the preparation processes for model training, and their execution order is not sequential. It only needs to be executed before model training.

[0074] S103: GRU model training: Input the historical log data in the log sample dataset into the constructed GRU model for learning and training. Determine the model loss and adjust the model parameters according to the classification result output by the model and the classification label in the log sample dataset. After multiple iterative trainings, obtain a trained GRU model with the model loss meeting the requirements.

[0075] When training the GRU model, use the pre-constructed sample dataset as the input of the model. The output of the model is simple log data or complex log data. Based on the classification result output by the model and the classification label in the log sample dataset, adjust the model parameters based on the loss function of the GRU model to reach a relatively accurate regression state to improve the GRU model and minimize the loss. When performing model training, the historical log data can be divided into multiple subsets, and each subset is input for training each time. Calculate the model loss based on the preset loss function according to the classification result output by the model and the classification label in the log sample dataset. If the model loss does not meet the expected requirements, adjust the model parameters and perform the next round of training. After multiple iterative trainings, until the model loss meets the expected requirements, obtain a trained CRU model.

[0076] S104: Integrate and deploy the trained GRU model and the CMDB system into the log processing system.

[0077] Among them, the GRU model is used to obtain the log data to be screened or historical log data from the log processing system, and feedback the classified result of the output log data to the log processing system; the CMDB system is used to store configuration items. The integration of the log processing system and the CMDB system can be implemented using the API interface provided by the CMDB system.

[0078] This step deploys the GRU model to the actual log processing system for real-time screening of log data to screen out complex data; this step is an optional step. If no integration is performed, the GRU model can also be set in a separate device. After the classification result of the GRU model is provided to the log data processing system, the log data processing system will then perform operations such as association and structured processing of CMDB configuration items according to the classification result.

[0079] S105: Use the trained GRU model to classify the log data to be screened, and determine whether the log data to be screened is complex log data or simple log data;

[0080] Before performing structured log data, the log data is fully screened, and the log data to be structured is used as the log data to be screened. The complex log data is screened out from the log data to be screened through the GRU model for structured processing, avoiding structuring simple data and wasting time and resources. When screening, the log vector data of the candidate log data is used as the input of the GRU model. For the generation process of the log vector data, refer to the above relevant description.

[0081] S106: According to the unique identifier and configuration information included in the CMDB configuration items, perform association and clustering analysis on the determined complex log data, and perform structured processing on the complex log data according to the association and clustering analysis results.

[0082] Associate the screened complex log data with the configuration items in the CMDB, and create or update records in the CMDB to reflect the status or attributes of the configuration items related to the log events, completing the structuring of the complex log data. Specifically, according to the unique identifier included in the CMDB configuration items, it can be determined whether the devices or components involved in the complex log data are the devices or components already recorded in the CMDB system; if not, an unregistered reminder message is sent; if so, according to the configuration item corresponding to the unique identifier, the associated devices, components, and historical log data are determined; according to the associated devices, components, and historical log data, the complex log data is clustered; based on the clustering processing result, the structured data of the complex log data is obtained.

[0083] Optionally, the above method further includes updating the CMDB configuration items according to the identifier and configuration information included in the determined complex log data.

[0084] The above steps S101 - S104 are the preparation processes for log structuring, and are not processes that need to be executed before structuring each log. After the model is trained, it can be integrated into the log processing system for processing the generated logs.

[0085] Based on the same inventive concept, an embodiment of the present invention further provides a log data processing device, which can be set in a computer device with data processing capabilities. The structure of the device is as Figure 2 shown, including:

[0086] A log classification module 11, configured to classify the log data to be screened using a trained gated recurrent unit (GRU) model, and determine whether the log data to be screened is complex log data or simple log data;

[0087] A data processing module 12, configured to classify and label the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex;

[0088] A model training module 13, configured to input the historical log data in the log sample data set into the constructed GRU model for learning and training, determine the model loss according to the classification result output by the model and the classification labels in the log sample data set, and adjust the model parameters. After multiple iterative trainings, a trained GRU model with a model loss meeting the requirements is obtained.

[0089] Optionally, the above device further includes:

[0090] A structuring processing module 14, configured to perform association and clustering analysis on the determined complex log data according to the unique identifier and configuration information included in the CMDB configuration item, and perform structuring processing on the complex log data according to the association and clustering analysis results.

[0091] Based on the same inventive concept, an embodiment of the present invention further provides a log data processing system, the structure of which is as Figure 3 shown. The CMDB system 21 and the GRU model 22 are integrated in the log data processing system 20;

[0092] The GRU model 22 is used to classify the log data to be screened generated in the log data processing system, and determine whether the log data to be screened is complex log data or simple log data; the process of obtaining the trained GRU model includes: classifying and labeling the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex; inputting the historical log data in the log sample data set into the constructed GRU model for learning and training, determining the model loss and adjusting the model parameters according to the classification results output by the model and the classification labels in the log sample data set, and after multiple iterative trainings, obtaining a trained GRU model with the model loss meeting the requirements.

[0093] The CMDB system 21 is used to store CMDB configuration items;

[0094] The log data processing system 20 is used to perform association and clustering analysis on the determined complex log data according to the classification results of the GRU model and the CMDB configuration items, and perform structured processing on the complex log data according to the association and clustering analysis results.

[0095] An embodiment of the present invention further provides a computer storage medium, in which computer executable instructions are stored, and when the computer executable instructions are executed by a processor, the above-mentioned CMDB-based log data processing method is implemented.

[0096] An embodiment of the present invention further provides a log data processing device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the above-mentioned CMDB-based log data processing method is implemented.

[0097] Regarding the CMDB-based log data processing device and system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0098] The above methods, devices, and systems in the embodiments of the present invention fully screen the log data before structuring the log data, only structure the complex data, and do not structure the simple data, avoiding waste of time and resources, and being more conducive to subsequent log auditing and query work.

[0099] Unless otherwise specifically stated, terms such as "processing", "computing", "calculating", "determining", "displaying", etc. can refer to the actions and / or processes of one or more processing or computing systems, or similar devices, which operate on and transform data represented as physical (such as electronic) quantities within the registers or memories of the processing system into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals can be represented using any of a variety of different technologies and methods. For example, the data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0100] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in a process can be rearranged without departing from the scope of the present disclosure. The appended method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy recited.

[0101] In the above detailed description, various features are combined in a single embodiment to simplify the present disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention lies in less than the full scope of the features of the single disclosed embodiment. Accordingly, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0102] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability of hardware and software, the above description of the various illustrative components, blocks, modules, circuits, and steps has been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in a flexible manner for each particular application, but such implementation decisions should not be construed as departing from the scope of the present disclosure.

[0103] The steps of the methods or algorithms described in connection with the embodiments of this specification may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. The software modules may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also exist as discrete components in the user terminal.

[0104] For a software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit may be implemented within the processor or outside the processor, and in the latter case, it is communicatively coupled to the processor by means of various methods, which are well known in the art.

[0105] The above description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but those of ordinary skill in the art should recognize that the various embodiments can be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, this term is covered in a manner similar to the term "including," as "including" is interpreted when used as a transitional word in a claim. In addition, any use of the term "or" in the specification or claims of the patent application is intended to mean "non-exclusive or."

Claims

1. A method for processing log data based on CMDB, characterized in that, Including: Using a trained gated recurrent unit (GRU) model to classify the log data to be screened, and determining whether the log data to be screened is complex log data or simple log data; Structurally processing the complex log data in combination with the CMDB configuration items; The process of obtaining the trained GRU model includes: Classifying and labeling the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex; Inputting the historical log data in the log sample data set into the constructed GRU model for learning and training, determining the model loss and adjusting the model parameters according to the classification result output by the model and the classification labels in the log sample data set, and after multiple iterative trainings, obtaining a trained GRU model with the model loss meeting the requirements.

2. The method according to claim 1, characterized in that, The classifying and labeling the obtained historical log data to obtain a log sample data set includes: Obtaining historical log data from the operating system or application program; Classifying the historical log data according to the set log classification rules, determining whether the historical log data belongs to complex log data or simple log data, and adding a classification identifier to the historical log data according to the classification result; the log classification rules are set according to the log volume, the number of examples, and the complexity of the Configuration Management Database (CMDB); Converting the historical log data into log vector data through a word embedding text feature extraction method, and constructing a log sample data set including the log vector data and corresponding classification labels.

3. The method according to claim 1, wherein The converting the historical log data into log vector data through a word embedding text feature extraction method includes: For each historical log data, mapping the word sequence in the historical log data to the corresponding word embedding vector; Averaging or weighted averaging the word embedding vectors of each word sequence to obtain a vector representation of the historical log data; The vector representation of the historical log data matches the input format of the input layer of the GRU model.

4. The method according to claim 1, wherein The GRU model includes an input layer, a GRU layer, and an output layer; The input layer is used for inputting log vector data, the GRU layer is used for processing log vector data and extracting feature data of the log vector data, and the output layer is used for outputting a log classification result based on the feature data of the log vector data.

5. The method according to claim 1, characterized in that, It also includes: Integrating and deploying the trained GRU model and the CMDB system into the log processing system; Among them, the GRU model is used to obtain the log data to be screened or historical log data from the log processing system, and feedback the output log data classification result to the log processing system; the CMDB system is used to store CMDB configuration items.

6. The method according to claim 1, wherein The structurally processing the complex log data in combination with the CMDB configuration items includes: According to the unique identifier and configuration information included in the CMDB configuration items, performing association and clustering analysis on the determined complex log data, and structurally processing the complex log data according to the association and clustering analysis results.

7. The method according to claim 6, characterized in that, According to the unique identifier and configuration information included in the CMDB configuration items, performing association and clustering analysis on the determined complex log data includes: Determine whether the devices or components involved in the complex log data are the devices or components recorded in the CMDB system according to the unique identifier included in the CMDB configuration item; If not, send an unregistered reminder message; if so, determine the associated devices, components, and historical log data according to the configuration item corresponding to the unique identifier; cluster the complex log data according to the associated devices, components, and historical log data; based on the clustering result, obtain the structured data of the complex log data.

8. The method according to claim 6, wherein It further includes: Update the CMDB configuration item according to the identifier and configuration information included in the determined complex log data.

9. A log data processing device based on CMDB, characterized in that, It includes: A log classification module, which is used to classify the log data to be screened using a trained gated recurrent unit (GRU) model to determine whether the log data to be screened is complex log data or simple log data; A data processing module, which is used to classify and label the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex; A model training module, which is used to input the historical log data in the log sample data set into the constructed GRU model for learning and training, determine the model loss and adjust the model parameters according to the classification result output by the model and the classification labels in the log sample data set, and after multiple iterative trainings, obtain a trained GRU model with a model loss meeting the requirements.

10. The device according to claim 9, characterized in that, It further includes: A structured processing module, which is used to perform association and clustering analysis on the determined complex log data according to the unique identifier and configuration information included in the CMDB configuration item, and perform structured processing on the complex log data according to the association and clustering analysis results.

11. A log data processing system based on CMDB, characterized in that, The log data processing system integrates a CMDB system and a GRU model; The GRU model is used to classify the log data to be screened generated in the log data processing system to determine whether the log data to be screened is complex log data or simple log data; The process of obtaining the trained GRU model includes: classifying and labeling the obtained historical log data to obtain a log sample data set, where the log sample data set includes historical log data and corresponding classification labels, and the classification labels include simple and complex; inputting the historical log data in the log sample data set into the constructed GRU model for learning and training, determining the model loss and adjusting the model parameters according to the classification result output by the model and the classification labels in the log sample data set, and after multiple iterative trainings, obtaining a trained GRU model with a model loss meeting the requirements; The CMDB system is used to store CMDB configuration items; The log data processing system is used to perform association and clustering analysis on the determined complex log data according to the classification result of the GRU model and the CMDB configuration item, and perform structured processing on the complex log data according to the association and clustering analysis results.

12. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they implement the CMDB-based log data processing method according to any one of claims 1-8.

13. A log data processing device, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method for processing log data based on CMDB according to any one of claims 1-8 is implemented.

Images