A log processing method, device, equipment and readable storage medium

Abstract

The application discloses a log processing method, device and equipment and a readable storage medium in the computer technical field. The method disclosed by the application comprises the following steps: acquiring homologous IP logs belonging to the same customer, and determining word embedding features corresponding to each alarm type in the homologous IP logs; determining coding features of each alarm type in the homologous IP logs; and performing clustering processing on each alarm type in the homologous IP logs based on the word embedding features and the coding features of each alarm type in the homologous IP logs. The word embedding features can reflect the co-occurrence relationship between different alarm types in the homologous IP logs, and the coding features can reflect the features of each alarm type itself. The clustering processing of each alarm type in combination with the multi-dimensional feature information can improve the accuracy and efficiency of log classification. The log processing device, equipment and readable storage medium provided by the application also have the above technical effects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a log processing method and device, equipment and a readable storage medium. BACKGROUND

[0002] At present, security devices such as firewalls, intrusion detection systems and honeypots record various alarm types. These logs often alarm different devices, and the data volume is huge. In order to facilitate the analysis of these alarm types, technical personnel need to query a pre-defined log classification table, so as to classify similar logs or repeated logs together. This process depends on the experience of technical personnel, and there is a certain time lag, low accuracy and low efficiency.

[0003] Therefore, how to improve the classification accuracy and efficiency of alarm types is a problem to be solved by those skilled in the art. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a log processing method, device, equipment and readable storage medium to improve the classification accuracy and efficiency of alarm types. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a log processing method, comprising:

[0006] obtaining homologous IP logs belonging to the same customer, and determining the word embedding features corresponding to each alarm type in the homologous IP logs; the word embedding features corresponding to different alarm types having a co-occurrence relationship in the homologous IP logs are similar;

[0007] determining the coding features of each alarm type in the homologous IP logs;

[0008] based on the word embedding features and coding features of each alarm type in the homologous IP logs, performing clustering processing on each alarm type in the homologous IP logs.

[0009] Optionally, the determination of the word embedding features corresponding to each alarm type in the homologous IP logs comprises:

[0010] taking the alarm identification information of each alarm type in the homologous IP logs as a vocabulary, and constructing a to-be-recognized sentence;

[0011] inputting the to-be-recognized sentence into a word embedding model, so that the word embedding model outputs the word embedding features of each alarm type in the homologous IP logs.

[0012] Optionally, the determination of the coding features of each alarm type in the homologous IP logs comprises:

[0013] For each alarm type in the homologous IP log, each component of any alarm type is encoded respectively, and the encoding results of each component are spliced to obtain the encoding feature of the alarm type.

[0014] Optionally, each alarm type in the homologous IP log includes alarm identification information and alarm description information, and the encoding of each component of any alarm type includes:

[0015] For any alarm type, the alarm identification information is encoded by a normalization method, and the alarm description information is encoded by a word segmentation statistical method.

[0016] Optionally, the encoding of the alarm description information by the word segmentation statistical method includes:

[0017] The alarm description information is subjected to word segmentation processing.

[0018] The statistical information of each word in the word segmentation processing result appearing in a preset word library is determined, and the encoding result of the alarm description information is determined based on the statistical information.

[0019] Optionally, the generation process of the preset word library includes:

[0020] The alarm description information of each alarm type in the homologous IP log is subjected to word segmentation processing.

[0021] Words with an appearance frequency less than a preset threshold in the word segmentation processing result are filtered out, and the remaining words are used to construct the preset word library.

[0022] Optionally, the clustering processing of each alarm type in the homologous IP log based on the word embedding feature and the encoding feature of each alarm type in the homologous IP log includes:

[0023] The word embedding feature and the encoding feature of the same alarm type are spliced to obtain the splicing feature of the corresponding alarm type, and the splicing features of each alarm type in the homologous IP log are subjected to clustering processing.

[0024] Optionally, if any alarm type does not include alarm description information, or each word in the word segmentation processing result of the alarm description information does not appear in the preset word library, the mean value of the encoding results of other alarm description information is taken as the encoding result of the alarm description information of the current alarm type.

[0025] In a second aspect, the present application provides a log processing device, which includes:

[0026] The word embedding feature determination module is configured to acquire homologous IP logs belonging to the same customer, and determine word embedding features corresponding to each alarm type in the homologous IP logs; the word embedding features corresponding to different alarm types having a co-occurrence relationship in the homologous IP logs are similar;

[0027] The encoding feature determination module is configured to determine an encoding feature of each alarm type in the homologous IP logs.

[0028] The clustering processing module is configured to perform clustering processing on each alarm type in the homologous IP logs based on the word embedding feature and the encoding feature of each alarm type in the homologous IP logs.

[0029] In a third aspect, the present application provides an electronic device, comprising:

[0030] The memory is configured to store a computer program.

[0031] The processor is configured to execute the computer program to implement the log processing method disclosed above.

[0032] In a fourth aspect, the present application provides a readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the log processing method disclosed above.

[0033] According to the above scheme, the present application provides a log processing method, comprising: acquiring homologous IP logs belonging to the same customer, and determining word embedding features corresponding to each alarm type in the homologous IP logs; the word embedding features corresponding to different alarm types having a co-occurrence relationship in the homologous IP logs are similar; determining an encoding feature of each alarm type in the homologous IP logs; and performing clustering processing on each alarm type in the homologous IP logs based on the word embedding feature and the encoding feature of each alarm type in the homologous IP logs.

[0034] As can be seen, this application can identify co-occurrence of IP logs belonging to the same customer. Generally, co-occurrence exists between co-occurrence of co-occurrence of IP logs belonging to the same customer. To represent this co-occurrence, this application utilizes word embedding technology in the field of natural language processing to determine the word embedding features corresponding to each alarm type in the co-occurrence of IP logs. Thus, any word embedding feature can reflect the co-occurrence relationship between different alarm types in the co-occurrence of IP logs; that is, the word embedding features corresponding to different alarm types with co-occurrence relationships in the co-occurrence of IP logs are similar, while the word embedding features corresponding to different alarm types without co-occurrence relationships are dissimilar. Because word embedding features, when used in the field of natural language processing, can represent the co-occurrence relationship between words in a sentence, this application, based on a similar principle, uses word embedding features to represent the co-occurrence relationship between different alarm types in the co-occurrence of IP logs. This application then determines the encoding features of each alarm type in the same-source IP logs. Based on the word embedding features and encoding features of each alarm type in the same-source IP logs, clustering is performed on each alarm type in the same-source IP logs. This allows for the clustering of duplicate and similar logs together. This process requires no manual intervention and can improve the efficiency of classifying similar or duplicate logs. Simultaneously, word embedding features can reflect the co-occurrence relationships between different alarm types in the same-source IP logs, while encoding features can reflect the characteristics of each alarm type itself. Combining multi-dimensional feature information to cluster each alarm type can improve the accuracy of log classification.

[0035] Correspondingly, the log processing apparatus, device, and readable storage medium provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a log processing method disclosed in this application;

[0038] Figure 2 This is a flowchart of another log processing method disclosed in this application;

[0039] Figure 3 This is a schematic diagram of a log processing device disclosed in this application;

[0040] Figure 4 This is a schematic diagram of an electronic device disclosed in this application;

[0041] Figure 5This is a schematic diagram of another electronic device disclosed in this application;

[0042] Figure 6 This is a schematic diagram of an application scenario to which this application applies. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Currently, to facilitate the analysis of various alarm types logged by security devices such as firewalls, technicians need to query predefined log classification tables to group similar or duplicate logs together. This process relies on the experience of technicians, has a certain time lag, and suffers from low accuracy and efficiency. Therefore, this application provides a log processing scheme that can improve the accuracy and efficiency of alarm type classification.

[0045] See Figure 1 As shown in the figure, this application discloses a log processing method, including:

[0046] S101. Obtain the same-source IP logs belonging to the same customer, and determine the word embedding features corresponding to each alarm type in the same-source IP logs; the word embedding features corresponding to different alarm types with co-occurrence relationships in the same-source IP logs are similar.

[0047] Logs from the same source IP belonging to the same customer can be obtained from the alarm type set. Each alarm type in the set originates from security devices such as firewalls, intrusion detection systems, and honeypots on different hosts. Each alarm type corresponds to several attributes, including: alarm type ID (usually represented as an array, e.g., 11089812), customer ID, alarm time, source IP, and destination IP. Based on these attributes, alarm types from the same enterprise customer (with the same customer ID) can be grouped together. Since there are multiple protected devices in the network of the same enterprise customer, alarm types from the same enterprise customer can be further subdivided according to the source IP, thus obtaining logs from the same source IP belonging to the same customer. By merging alarm types with the same customer ID and the same source IP together, an alarm type sequence for a single protected device can be obtained. Alarm type sequences can be recorded using alarm type IDs. For example, alarm type sequences could be: 17820303, 17820316, 11021127, 17980004, 11023402. As you can see, an alarm type sequence consists of the IDs of logs generated by multiple security devices (such as firewalls, intrusion detection systems, and honeypots) on a single host (i.e., the protected device).

[0048] It should be noted that there are often correlations between different alarm types corresponding to a single device. For example, when memory resources are scarce, CPU utilization may also be too high. Discovering such correlations can be beneficial for log classification. Therefore, this embodiment uses word embedding technology from the field of natural language processing to represent each alarm type in the same-source IP logs. The word embedding features corresponding to each alarm type in the same-source IP logs can then reflect the co-occurrence relationship between different alarm types in the same-source IP logs. Classifying logs based on these word embedding features helps to group alarm types with co-occurrence together. If two alarm type IDs always appear simultaneously in an alarm type sequence, then the two alarm types corresponding to these two alarm type IDs can be considered to have co-occurrence. Co-occurrence refers to the probability of two or more targets appearing together. Specifically, in this application, it refers to the probability of two or more alarm types appearing together in the same-source IP logs of the same customer.

[0049] In one implementation, determining the word embedding features corresponding to each alarm type in the same-source IP log includes: using the alarm identification information of each alarm type in the same-source IP log as vocabulary to construct a sentence to be identified; inputting the sentence to be identified into a word embedding model so that the word embedding model outputs the word embedding features of each alarm type in the same-source IP log.

[0050] In one example, assuming the alarm identifiers for each alarm type in the same-origin IP logs are 17820303, 17820316, 11021127, 17980004, and 11023402, these five alarm identifiers are treated as five words, resulting in the sentence to be identified: 17820303 17820316 11021127 17980004 11023402. This sentence is then input into a word embedding model, which can uncover co-occurrence relationships between different words and reflect these relationships in the word embedding features corresponding to each word. The word embedding model can be trained similarly to a natural language processing model, simply by changing the training samples to sequences of alarm types corresponding to different devices. Specifically, the word embedding model can be trained using the skip-gram method. Since no labels are needed for the samples during training, it is unsupervised training, resulting in high training efficiency.

[0051] S102. Determine the encoding characteristics of each alarm type in the same source IP log.

[0052] S103. Based on the word embedding features and encoding features of each alarm type in the same source IP log, perform clustering processing on each alarm type in the same source IP log.

[0053] To avoid losing information about each alarm type, this embodiment also encodes each alarm type in the same-source IP log, thereby determining the encoding characteristics of each alarm type in the same-source IP log. Subsequently, clustering is performed on each alarm type based on its word embedding features and encoding features. This helps to group logs with co-occurrence relationships, high similarity, and repetitions together, thus improving the accuracy and efficiency of alarm type classification.

[0054] Generally, an alarm type consists of alarm identification information and alarm description information. Therefore, the different components of an alarm type can be encoded separately, and then the codes of the different components can be concatenated to obtain the encoding features of the current alarm type. Specifically, the alarm identification information can be encoded using a normalization method, while the alarm description information can be encoded using the TF / IDF method.

[0055] As can be seen, this embodiment can identify co-occurring IP logs belonging to the same customer. Generally, co-occurrence exists among co-occurring IP logs belonging to the same customer. To represent this co-occurrence, this application utilizes word embeddings from the field of natural language processing to determine the word embedding features corresponding to each alarm type in the co-occurring IP logs. Thus, any word embedding feature can reflect the co-occurrence relationship between different alarm types in the co-occurring IP logs. Subsequently, this application determines the encoding features of each alarm type in the co-occurring IP logs. Then, based on the word embedding features and encoding features of each alarm type in the co-occurring IP logs, clustering is performed on each alarm type in the co-occurring IP logs. This allows duplicate and similar logs to be clustered together. This process requires no human intervention and can improve the efficiency of classifying similar or duplicate logs. Simultaneously, word embedding features can reflect the co-occurrence relationship between different alarm types in the co-occurring IP logs, and encoding features can reflect the characteristics of each alarm type itself. Combining multi-dimensional feature information to cluster each alarm type can improve the accuracy of log classification.

[0056] Based on the above embodiments, it should be noted that an alarm type generally consists of alarm identification information and alarm description information. For example, an alarm type like 13030147, Cross-Site Scripting Attack Detection, includes alarm identification information to mark the current alarm type and alarm description information to describe the attack characteristics of the alarm type. Therefore, the components of an alarm type include alarm identification information and alarm description information. In one implementation, determining the encoding characteristics of each alarm type in the same-source IP log includes: for each alarm type in the same-source IP log, encoding each component of any alarm type separately, and concatenating the encoding results of each component to obtain the encoding characteristics of that alarm type.

[0057] In one implementation, each alarm type in the same-origin IP log includes: alarm identification information and alarm description information. Each component of any alarm type is encoded separately, including: for any alarm type, encoding the alarm identification information using a normalization method and encoding the alarm description information using a word segmentation and statistical method. It is evident that the alarm identification information can be encoded using a normalization method, while the alarm description information can be encoded using a word segmentation and statistical method.

[0058] Alarm descriptions are typically a single sentence. Therefore, word segmentation methods such as jieba can be used to segment the alarm description first. Then, the frequency of each word in the segmented result is examined within a pre-defined vocabulary. Based on this information, the encoding result of the alarm description can be determined. In one implementation, alarm descriptions are encoded using a word segmentation statistical method, including: segmenting the alarm description; determining the statistical information of each word in the segmentation result appearing in a pre-defined vocabulary; and determining the encoding result of the alarm description based on the statistical information. If none of the words in the segmentation result of a certain alarm description A appear in the pre-defined vocabulary, or if a certain alarm type has no description information, then the average of the encoding results corresponding to other alarm descriptions is used as the encoding result of the current alarm description A.

[0059] In one implementation, the process of generating the preset vocabulary includes: performing word segmentation on the alarm description information of each alarm type in the same-origin IP logs; filtering out words whose frequency in the word segmentation results is less than a preset threshold, and using the remaining words to construct the preset vocabulary. Specifically, the TF / IDF method can be used to construct the preset vocabulary.

[0060] In this application, the word embedding features corresponding to different alarm types with co-occurrence relationships in the same-source IP logs are similar, while the encoding features can reflect the information of the alarm type itself. Therefore, the concatenated features obtained by splicing word embedding features and encoding features have the characteristics of both word embedding features and encoding features. It can reflect both the co-occurrence relationship between different alarm types in the same-source IP logs and the information of the alarm type itself. Therefore, clustering analysis of each alarm type based on the concatenated features can improve classification accuracy and make it easier to classify duplicate logs and similar logs together. Therefore, in one embodiment, clustering analysis is performed on each alarm type based on the word embedding features and encoding features of each alarm type in the same-source IP logs, including: splicing the word embedding features and encoding features of the same alarm type to obtain the concatenated features of the corresponding alarm type, and performing clustering analysis on the concatenated features of each alarm type.

[0061] The following embodiment performs feature transformation on each alarm type based on three dimensions, and then uses the mean-shift algorithm to cluster the feature vectors corresponding to each alarm type to group similar feature vectors into the same class. Subsequently, operation experts further analyze the alarm types in the same class, thereby quickly determining which logs need to be deleted and which need to be merged, improving the efficiency of log analysis.

[0062] This embodiment pre-collects alarm types recorded by various safety devices over a period of time (e.g., 24 hours) into a database, and then processes them in three steps. Please refer to the following for details. Figure 2 .

[0063] Step 1: Merge logs with the same customer ID and source IP in the database to obtain an alarm type sequence for a single host. Train a word embedding model using each alarm type sequence, and then use the word embedding model to obtain the vector representation of the word embedding features of each alarm type in the database. The word embedding model can mimic natural language processing models, treating each alarm type (e.g., 17820303) in the sequence as a separate word, and each alarm type sequence as a single sentence. This allows the word embedding model to discover the co-occurrence between different alarm types, ensuring that alarm types that frequently appear together in a sequence have similar word embedding features. An alarm type represents a specific alarm message.

[0064] Step 2: Normalize the alarm type ID (e.g., 17820303) to obtain a vector representation of each alarm type ID. Generally, the IDs for different alarm types tend to have some similarity in naming. Therefore, encoding the IDs as feature vectors can also measure the similarity between different alarm types. Specifically, the mean and variance of the IDs can be calculated, and the mean and variance can be normalized to 0 and 1 respectively. Finally, the normalized result is used as the vector representation of the ID.

[0065] Step 3: Perform Chinese and English word segmentation and TF / IDF feature representation on the alarm description information. Specifically, firstly, the jieba Chinese and English word segmentation algorithm is used to segment the Chinese and English alarm description information. Then, the TF / IDF model is used to encode the segmented words to obtain the final vector representation. It should be noted that words with a frequency of less than N can be filtered out, thereby limiting the size of the TF / IDF vocabulary. At the same time, if there is no alarm description information for an alarm type, or if its description information does not match the vocabulary, the average of the final vector representations of other alarm types is used as the final vector representation of the current alarm description information.

[0066] Based on the above three steps, we can obtain the word embedding features, the vector representation of the ID number, and the vector representation of the alarm description information. Then, by concatenating these three features, we can represent each alarm type using the concatenated feature representation. After obtaining the feature representation of each alarm type, we use the mean-shift algorithm for clustering to group alarm types with similar feature representations into the same cluster. Since the clustering process may suffer from under-clustering / over-clustering issues, it is recommended that operations experts conduct further analysis on each cluster to determine whether different alarm types within the same cluster can be merged.

[0067] Thus, this embodiment uses multi-dimensional information about alarm types to automatically aggregate alarm types with high similarity, eliminating the need to manually write log classification tables and bringing convenience to multi-source log analysis.

[0068] The following describes a log processing apparatus provided in an embodiment of this application. The log processing apparatus described below and the log processing method described above can be referred to each other.

[0069] See Figure 3 As shown in the figure, this application discloses a log processing apparatus, including:

[0070] The word embedding feature determination module 301 is used to obtain the same-source IP logs belonging to the same customer and determine the word embedding features corresponding to each alarm type in the same-source IP logs; the word embedding features corresponding to different alarm types with co-occurrence relationships in the same-source IP logs are similar;

[0071] The encoding feature determination module 302 is used to determine the encoding features of each alarm type in the same source IP log;

[0072] The clustering module 303 is used to perform clustering processing on each alarm type in the same source IP log based on the word embedding features and encoding features of each alarm type in the same source IP log.

[0073] In one implementation, the word embedding feature determination module is specifically used for:

[0074] Use alarm identifiers for each alarm type in the same-source IP logs as words to construct sentences to be identified;

[0075] Input the sentence to be identified into the word embedding model so that the word embedding model outputs word embedding features of each alarm type in the same source IP log.

[0076] In one implementation, the encoding feature determination module is specifically used for:

[0077] For each alarm type in the same-source IP log, each component of any alarm type is encoded separately, and the encoding results of each component are concatenated to obtain the encoding feature of that alarm type.

[0078] In one implementation, each alarm type in the same-source IP log includes: alarm identification information and alarm description information, and the encoding feature determination module is specifically used for:

[0079] For any alarm type, the alarm identification information is encoded using a normalization method, and the alarm description information is encoded using a word segmentation and statistical method.

[0080] In one implementation, the encoding feature determination module is specifically used for:

[0081] The alarm description information is segmented into words;

[0082] The statistical information of each word in the word segmentation result appears in the preset vocabulary library is determined, and the encoding result of the alarm description information is determined based on the statistical information.

[0083] In one implementation, if any alarm type does not include alarm description information, or if any word in the word segmentation result of the alarm description information does not appear in the preset vocabulary, then the average value of the encoding results of other alarm description information is used as the encoding result of the alarm description information of the current alarm type.

[0084] In one implementation, the process of generating a preset vocabulary database includes:

[0085] Perform word segmentation on the alarm description information of each alarm type in the same-origin IP log;

[0086] Filter out words that appear less than a preset threshold in the word segmentation results, and use the remaining words to build a preset vocabulary library.

[0087] In one implementation, the clustering processing module is specifically used for:

[0088] By concatenating the word embedding features and encoding features of the same alarm type, the concatenated features of the corresponding alarm type are obtained, and the concatenated features of each alarm type in the same source IP logs are clustered.

[0089] For more detailed information on the working process of each module and unit in this embodiment, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0090] As can be seen, this embodiment provides a log processing device that can improve the accuracy and efficiency of alarm type classification.

[0091] The following describes an electronic device provided in an embodiment of this application. The electronic device described below can be referred to in conjunction with the log processing method and apparatus described above.

[0092] See Figure 4 As shown in the figure, an embodiment of this application discloses an electronic device, including:

[0093] Memory 401 is used to store computer programs;

[0094] Processor 402 is configured to execute the computer program to implement the method disclosed in any of the above embodiments.

[0095] Please refer to Figure 5 , Figure 5This is a schematic diagram of another electronic device provided in this embodiment. This electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the electronic device 301.

[0096] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341. Examples include Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0097] exist Figure 5 In this context, application 342 may be a program that executes the log processing method, and data 344 may be data required or generated by executing the log processing method.

[0098] The steps in the log processing method described above can be implemented by the structure of an electronic device.

[0099] The following describes a readable storage medium provided in an embodiment of this application. The readable storage medium described below can be referred to in conjunction with the log processing method, apparatus and device described above.

[0100] A readable storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the log processing method disclosed in the foregoing embodiments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0101] The following describes the application scenarios and physical architecture to which this application applies. For example... Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the application scenario to which this application applies.

[0102] It should be understood that the log processing method provided in this application can be applied to systems or programs with log processing functions. Specifically, systems or programs with log processing functions can run on devices such as servers and personal computers.

[0103] exist Figure 6 In this system, a system or program with log processing capabilities can run on a server. This server obtains a set of alarm types from other terminal devices via the network. It identifies homologous IP logs belonging to the same customer within the alarm type set and determines the word embedding features corresponding to each alarm type in the homologous IP logs. The word embedding features corresponding to different alarm types with co-occurrence relationships in the homologous IP logs are similar. The encoding features of each alarm type in the homologous IP logs are determined. Based on the word embedding features and encoding features of each alarm type in the homologous IP logs, clustering processing is performed on each alarm type in the homologous IP logs.

[0104] like Figure 6 It is known that the server can establish communication connections with multiple devices, and obtain alarm types from these devices. The server collects the alarm types uploaded by these devices to form an alarm type set for subsequent processing.

[0105] Figure 6 The document shows various terminal devices. In real-world scenarios, more or fewer types of terminal devices may participate in the log processing process. The specific number and types depend on the actual scenario and are not limited here. Figure 6 The image shows one server, but in real-world scenarios, multiple servers can be involved, with the specific number depending on the actual situation.

[0106] It should be noted that the log processing method provided in this embodiment can be performed offline, that is, the server has a set of alarm types stored locally, and can directly use the solution provided in this application to process logs.

[0107] It is understandable that the aforementioned systems and programs with log processing capabilities can also run on personal mobile terminals or as a type of cloud service program. The specific operating mode depends on the actual scenario and is not limited here. Specifically, after the logs are processed, the obtained clustering results can be used to analyze and classify the logs.

[0108] The terms “first,” “second,” “third,” “fourth,” etc., used in this application (if applicable) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, or apparatus.

[0109] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0110] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of readable storage medium known in the art.

[0112] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A log processing method characterized by, The method comprises: obtaining homologous IP logs belonging to the same customer, and determining word embedding features corresponding to each alarm type in the homologous IP logs; word embedding features corresponding to different alarm types having a co-occurrence relationship in the homologous IP logs are similar; wherein the word embedding features corresponding to each alarm type in the homologous IP logs are determined by using a word embedding technology in the field of natural language processing, and any word embedding feature can reflect the co-occurrence relationship between different alarm types in the homologous IP logs; determining coding features of each alarm type in the homologous IP logs; performing clustering processing on each alarm type in the homologous IP logs based on the word embedding features and the coding features of each alarm type in the homologous IP logs; wherein the determination of the coding features of each alarm type in the homologous IP logs comprises: for each alarm type in the homologous IP logs, encoding each component of any alarm type respectively, and splicing the coding results of each component to obtain the coding features of the alarm type.

2. The log processing method of claim 1, wherein, The determination of the word embedding features corresponding to each alarm type in the homologous IP logs comprises: taking alarm identification information of each alarm type in the homologous IP logs as a vocabulary to construct a to-be-recognized sentence; inputting the to-be-recognized sentence into a word embedding model to enable the word embedding model to output the word embedding features of each alarm type in the homologous IP logs.

3. The log processing method of claim 1, wherein, Each alarm type in the homologous IP logs comprises alarm identification information and alarm description information. Correspondingly, the encoding of each component of any alarm type comprises: for any alarm type, encoding the alarm identification information by a normalization method and encoding the alarm description information by a word segmentation statistical method.

4. The log processing method of claim 3, wherein, The encoding of the alarm description information by the word segmentation statistical method comprises: performing word segmentation processing on the alarm description information; determining statistical information of each vocabulary in the word segmentation processing result appearing in a preset vocabulary library, and determining the coding result of the alarm description information based on the statistical information.

5. The log processing method of claim 4, wherein, The generation process of the preset vocabulary library comprises: performing word segmentation processing on the alarm description information of each alarm type in the homologous IP logs; filtering out vocabularies with an appearance frequency less than a preset threshold in the word segmentation processing result, and constructing the preset vocabulary library by using the remaining vocabularies.

6. The log processing method according to claim 3, wherein if any alarm type does not include alarm description information, or each vocabulary in the word segmentation processing result of the alarm description information does not appear in a preset vocabulary library, an average value of coding results of other alarm description information is taken as the coding result of the alarm description information of the current alarm type. The clustering processing on each alarm type in the homologous IP logs based on the word embedding features and the coding features of each alarm type in the homologous IP logs comprises:

7. The log processing method according to any one of claims 1 to 6, characterized in that, splicing the word embedding features and the coding features of the same alarm type to obtain splicing features of the corresponding alarm type, and performing clustering processing on the splicing features of each alarm type in the homologous IP logs. The method comprises:

8. A log processing apparatus characterized by comprising: ​ The word embedding feature determination module is configured to acquire homologous IP logs belonging to the same customer, and determine word embedding features corresponding to different alarm types in the homologous IP logs. The word embedding features corresponding to different alarm types having a co-occurrence relationship in the homologous IP logs are similar. The word embedding features corresponding to different alarm types in the homologous IP logs are determined by using a word embedding technology in the field of natural language processing, and any word embedding feature can reflect the co-occurrence relationship between different alarm types in the homologous IP logs. The encoding feature determination module is configured to determine an encoding feature of each alarm type in the homologous IP logs. The clustering processing module is configured to perform clustering processing on different alarm types in the homologous IP logs based on the word embedding features and the encoding features of each alarm type in the homologous IP logs. The encoding feature determination module is specifically configured to:

9. An electronic device, comprising: For each alarm type in the homologous IP logs, encode each component part of any alarm type respectively, and splice the encoding results of the component parts to obtain the encoding feature of the alarm type. The computer program is stored in the memory and executed by the processor to implement the log processing method according to any one of claims 1 to 7. The computer program is stored in the memory and executed by the processor to implement the log processing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized by, The computer program is stored in the memory and executed by the processor to implement the log processing method according to any one of claims 1 to 7.

Images