A log processing method, device, equipment and medium

Abstract

The present application discloses a log processing method, device, equipment, and medium, which includes: obtaining logs to be classified; extracting features from each log in the logs to be classified to obtain a set of log feature items corresponding to each log in the logs to be classified; determining a log vector corresponding to each log in the logs to be classified based on the set of log feature items corresponding to each log in the logs to be classified; and classifying the log vectors corresponding to each log in the logs to be classified using an ant colony clustering algorithm to classify the logs to be classified. In this way, the logs can be classified, and the accuracy and consistency of the classification results are improved, and the applicability is strong.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log processing method, device, equipment, and medium. Background Art

[0002] Cluster analysis is an important branch in the field of data mining. Clustering is to group data objects into multiple classes or clusters. Objects in the same cluster have high similarity, while objects in different clusters are quite different. Existing clustering algorithms are mainly divided into four categories: partitioning method, hierarchical method, density-based method and grid-based method.

[0003] The inventors have found that the above existing technologies may have the following problems: first, the above existing technologies require users to provide certain clustering prior information, which makes the clustering results very sensitive to the input parameters, which greatly reduces the adaptability of the classification method. Second, the above existing technologies are currently based on heuristic mechanism algorithms. Such methods have high solution efficiency, but are often prone to fall into local optimality, making it difficult to ensure the accuracy and consistency of the clustering results. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a log processing method, device, equipment, and medium that can classify logs and improve the accuracy and consistency of classification results, and has strong applicability. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a log processing method, comprising:

[0006] Get the logs to be classified;

[0007] Extracting feature items from each log in the logs to be classified to obtain a set of log feature items corresponding to each log in the logs to be classified;

[0008] Determine the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified;

[0009] The log vectors corresponding to the logs to be classified are classified by using an ant colony clustering algorithm, so as to classify the logs to be classified.

[0010] Optionally, after obtaining the logs to be classified, the method further includes:

[0011] Obtain the log classification parameters corresponding to the log to be classified.

[0012] Optionally, the classifying the log vectors corresponding to each log in the log to be classified by using an ant colony clustering algorithm includes:

[0013] A01: determining an initial cluster center from the log vector, and determining pheromones from the log vectors to be classified except the initial cluster center in the log vector to each of the initial cluster centers;

[0014] A02: dividing each of the log vectors to be classified into a class corresponding to the initial cluster center based on the pheromone;

[0015] A03: Determine the sum of distances from each log vector to other log vectors except itself;

[0016] A04: updating the initial cluster center based on the distance and the pheromone;

[0017] Step A02 is executed again until the cluster center after updating is the same as the cluster center before updating, or the current number of iterations is equal to the preset maximum number of iterations, and the classification of the log vector is completed.

[0018] Optionally, the determining of pheromones from the log vectors to be classified except the initial clustering center in the log vectors to the initial clustering centers includes:

[0019] Based on the first operation formula, the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers is determined, wherein the first operation formula is:

[0020]

[0021] Among them, τ ij represents the pheromone from the i-th log vector to be classified to the j-th initial cluster center, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th initial cluster center, and r represents the preset cluster center radius.

[0022] Optionally, the dividing each of the to-be-classified log vectors into a class corresponding to the initial cluster center based on the pheromone includes:

[0023] Determine the probability of classifying each of the to-be-classified log vectors into the class corresponding to each of the initial cluster centers based on the pheromone and the second operation formula;

[0024] According to the probability, each of the log vectors to be classified is divided into the class corresponding to the initial cluster center, wherein the second operation formula is:

[0025]

[0026] Among them, the P ijIt represents the probability of classifying the i-th log vector to be classified into the class corresponding to the j-th initial clustering center. α and β are both preset adjustment factors. S represents the set of log vectors to be classified whose Euclidean distance to the j-th initial clustering center is less than or equal to the preset clustering center radius.

[0027] Optionally, determining the sum of distances from each log vector to other log vectors other than itself includes:

[0028] The sum of the distances of each log vector to other log vectors other than itself is determined based on a third operation formula, wherein the third operation formula is:

[0029]

[0030]

[0031] Among them, L m represents the sum of the distances from the mth log vector to other log vectors except itself, x m represents the mth log vector, x mp is the pth value of the mth log vector, N represents the total number of log vectors, c mp is the transition vector c m The pth value in ||x m -c m || 2 represents the log vector x m and the transition vector c m The square of the modulus.

[0032] Optionally, the updating of the initial cluster center based on the distance and the updating of the pheromone includes:

[0033] Determine the minimum distance and the corresponding log vector as a new cluster center, and use the new cluster center to update the initial cluster center;

[0034] The pheromone is updated using a fourth operation formula, wherein the fourth operation formula is:

[0035]

[0036] Among them, τ ij ′ represents the pheromone of the i-th log vector to be classified to the j-th cluster center after updating, τ ij represents the pheromone from the i-th log vector to be classified to the j-th cluster center before updating, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th cluster center, ρ represents the volatility of the pheromone, and Q represents the total amount of preset pheromone.

[0037] In a second aspect, the present application discloses a log processing device, comprising:

[0038] A data acquisition module is used to obtain logs to be classified;

[0039] A feature item extraction module is used to extract feature items from each log in the log to be classified to obtain a set of log feature items corresponding to each log in the log to be classified;

[0040] A log vector determination module, used to determine the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified;

[0041] The log classification module is used to classify the log vectors corresponding to each log in the log to be classified by using an ant colony clustering algorithm, so as to classify the log to be classified.

[0042] In a third aspect, the present application discloses an electronic device, including:

[0043] Memory and processor;

[0044] Wherein, the memory is used to store computer programs;

[0045] The processor is used to execute the computer program to implement the aforementioned disclosed log processing method.

[0046] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed log processing method when executed by a processor.

[0047] It can be seen that the present application first obtains the log to be classified, and extracts the feature items of each log in the log to be classified, obtains the log feature item set corresponding to each log in the log to be classified, and then determines the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified, and then the ant colony clustering algorithm can be used to classify the log vector corresponding to each log in the log to be classified, so as to classify the log to be classified. In this way, after obtaining the log to be classified, feature extraction and other processing are performed on the obtained log to be classified to obtain the corresponding log vector, and then the ant colony clustering algorithm is used to classify each log vector, so that the obtained log to be classified can be classified. Compared with the prior art that requires the input of prior information, the ant colony clustering algorithm does not require the setting of prior information, thereby reducing the burden on users, improving clustering results, and improving applicability. The essence of the ant colony clustering algorithm is a very effective random search mechanism, rather than a heuristic mechanism, thereby improving the accuracy and consistency of the classification results. In addition, the algorithm structure and operation of the ant colony clustering algorithm are relatively simple and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A flow chart of a log processing method disclosed in this application;

[0050] Figure 2 A partial flow chart of a specific log processing method disclosed in this application;

[0051] Figure 3 A flow chart of a specific log processing method disclosed in this application;

[0052] Figure 4 This is a structural schematic diagram of a log processing device disclosed in this application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] See also Figure 1 As shown, the embodiment of the present application discloses a log processing method, which includes:

[0056] Step S11: Obtain logs to be classified.

[0057] In the specific implementation process, it is necessary to first obtain the logs to be classified, wherein the logs to be classified include multiple logs. Specifically, the logs to be classified can be obtained from the access log library of the website, etc. The specific way of obtaining the logs to be classified is not limited here.

[0058] After the log to be classified is obtained, it is also necessary to obtain the log classification parameters corresponding to the log to be classified, wherein the log classification parameters include but are not limited to a preset cluster center radius and the like.

[0059] Step S12: extracting feature items from each log in the log to be classified to obtain a set of log feature items corresponding to each log in the log to be classified.

[0060] After the logs to be classified are obtained, the logs to be classified need to be processed accordingly, so as to obtain the log vectors corresponding to the logs to be classified, so that the corresponding devices can perform classification processing.

[0061] Specifically, it is necessary to extract feature items from each log in the log to be classified to obtain a set of log feature items corresponding to each log in the log to be classified, wherein the feature items that need to be extracted from each log include but are not limited to excessive outbound traffic, excessive inbound traffic, VPN login during off-get off work hours, firewall acceptance, firewall rejection, login from outside the internal network, multiple consecutive failed logins, at least one successful login, a single source detecting multiple target IPs, and a single source detecting multiple target IPs and ports.

[0062] Step S13: determining a log vector corresponding to each log in the log to be classified based on a set of log feature items corresponding to each log in the log to be classified.

[0063] After obtaining the log feature item set corresponding to each log in the log to be classified, the log vector corresponding to each log in the log to be classified can be determined based on the log feature item set corresponding to each log in the log to be classified.

[0064] Specifically, for the current feature item corresponding to any log, if the feature item is extracted, the value corresponding to the feature item is represented as 1, and if the feature item is not extracted, the value corresponding to the feature item is represented as 0, and the log vector corresponding to the log is obtained.

[0065] For example, for log A, excessive outbound traffic, excessive inbound traffic, off-hours VPN login, firewall acceptance, and firewall rejection are extracted from log A, but login from outside the internal network, multiple consecutive failed logins, at least one successful login, multiple target IPs from a single source, and multiple target IPs and ports from a single source are not extracted. The log vector corresponding to log A is represented as (1, 1, 1, 1, 1, 0, 0, 0, 0, 0).

[0066] Step S14: using an ant colony clustering algorithm to classify the log vectors corresponding to each log in the log to be classified, so as to classify the log to be classified.

[0067] After obtaining the log vector corresponding to each log in the log to be classified, the log vector corresponding to each log in the log to be classified may be classified using an ant colony clustering algorithm to classify the log to be classified.

[0068] The foraging process of ants can be divided into two parts: searching for food and carrying food. Each ant releases pheromones on the path it passes during movement, and can sense the pheromones and their intensity. The more ants pass through the path, the stronger the pheromone is, and the pheromone itself will evaporate over time. Ants tend to move in the direction of high pheromone intensity. The more ants walk on a path, the greater the probability that later ants will choose this path. The behavior of the entire ant colony shows a positive feedback phenomenon. The basic idea of ​​the ant colony clustering algorithm is as follows: the data is regarded as ants with different attributes, and the cluster center is the "food source" that the ants are looking for. Then the data clustering process can be regarded as the process of ants looking for food sources.

[0069] It can be seen that the present application first obtains the log to be classified, and extracts the feature items of each log in the log to be classified, obtains the log feature item set corresponding to each log in the log to be classified, and then determines the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified, and then the ant colony clustering algorithm can be used to classify the log vector corresponding to each log in the log to be classified, so as to classify the log to be classified. In this way, after obtaining the log to be classified, feature extraction and other processing are performed on the obtained log to be classified to obtain the corresponding log vector, and then the ant colony clustering algorithm is used to classify each log vector, so that the obtained log to be classified can be classified. Compared with the prior art that requires the input of prior information, the ant colony clustering algorithm does not require the setting of prior information, thereby reducing the burden on users, improving clustering results, and improving applicability. The essence of the ant colony clustering algorithm is a very effective random search mechanism, rather than a heuristic mechanism, thereby improving the accuracy and consistency of the classification results. In addition, the algorithm structure and operation of the ant colony clustering algorithm are relatively simple and easy to implement.

[0070] See also Figure 2 As shown, the log vectors corresponding to each log in the log to be classified are classified using the ant colony clustering algorithm, which may specifically include:

[0071] A01: determining an initial cluster center from the log vector, and determining pheromones from the log vectors to be classified except the initial cluster center in the log vector to each of the initial cluster centers;

[0072] A02: dividing each of the log vectors to be classified into a class corresponding to the initial cluster center based on the pheromone;

[0073] A03: Determine the sum of distances from each log vector to other log vectors except itself;

[0074] A04: updating the initial cluster center based on the distance and the pheromone;

[0075] Step A02 is executed again until the cluster center after updating is the same as the cluster center before updating, or the current number of iterations is equal to the preset maximum number of iterations, and the classification of the log vector is completed.

[0076] Specifically, the log vectors corresponding to each log in the log to be classified are classified using the ant colony clustering algorithm. Initialization is first required, that is, a certain number of initial cluster centers are randomly determined from the log vectors, and then the pheromones of the log vectors to be classified other than the initial cluster centers in the log vectors are determined based on the first operation formula to each of the initial cluster centers, wherein the first operation formula is:

[0077]

[0078] Among them, τ ij represents the pheromone from the i-th log vector to be classified to the j-th initial cluster center, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th initial cluster center, and r represents the preset cluster center radius.

[0079] Then, based on the pheromone, each of the log vectors to be classified is divided into the class corresponding to the initial cluster center. Specifically, the probability of dividing each of the log vectors to be classified into the class corresponding to each of the initial cluster centers is first determined based on the pheromone and the second operation formula, and then each of the log vectors to be classified is divided into the class corresponding to the initial cluster center according to the probability, wherein the second operation formula is:

[0080]

[0081] Among them, the P ij represents the probability of classifying the i-th log vector to be classified into the class corresponding to the j-th initial cluster center, α and β are both preset adjustment factors, and S represents the set of log vectors to be classified whose Euclidean distance to the j-th initial cluster center is less than or equal to the preset cluster center radius. That is, firstly, based on the pheromone and the second operation formula, the probability of classifying each log vector to be classified into the class corresponding to each initial cluster center is determined, and then the current log vector to be classified is classified into the class corresponding to the initial cluster center corresponding to the maximum probability.

[0082] In the actual process, α and β can be 0.9 and 0.01 respectively. α and β can prevent the stagnant search caused by all ants getting the same result along the same path, and reproduce the idea of ​​the classic greedy algorithm.

[0083] For example, the initial cluster centers include log vector A and log vector B, the log vectors to be classified include log vector C, the probability of classifying log vector C into the class corresponding to log vector A is 0.7, and the probability of classifying log vector C into the class corresponding to log vector B is 0.3, then log vector C is classified into the class corresponding to log vector A.

[0084] When the log vectors to be classified are all classified into the classes corresponding to the corresponding initial clustering centers, the first round of clustering is completed, and the sum of the distances between each log vector and other log vectors except itself needs to be determined.

[0085] Specifically, the sum of the distances of each log vector to other log vectors other than itself is determined based on the third operation formula, wherein the third operation formula is:

[0086]

[0087]

[0088] Among them, L m represents the sum of the distances from the mth log vector to other log vectors except itself, x m represents the mth log vector, x mp is the pth value of the mth log vector, N represents the total number of log vectors, c mp is the transition vector c m The pth value in ||x m -c m || 2 represents the log vector x m and the transition vector c m The square of the modulus. In the third operation formula, c m is an intermediate transition vector.

[0089] It is also necessary to update the initial cluster center based on the distance and the pheromone. Specifically, the minimum distance and the corresponding log vector are first determined as the new cluster center, and the initial cluster center is updated using the new cluster center; and then the pheromone is updated using the fourth operation formula, wherein the fourth operation formula is:

[0090]

[0091] Among them, τ ij ′ represents the pheromone of the i-th log vector to be classified to the j-th cluster center after updating, τ ij represents the pheromone from the i-th log vector to be classified to the j-th cluster center before updating, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th cluster center, ρ represents the volatility of the pheromone, and Q represents the total amount of preset pheromone.

[0092] After determining the new cluster center, it can be determined whether the determined new cluster center is the same as the cluster center before the new cluster center is determined. If they are the same, it means that the cluster center tends to be stable and the classification is completed. If they are not the same, it is necessary to determine whether the current number of iterations is not less than the preset maximum number of iterations threshold. If the current number of iterations is less than the preset maximum number of iterations threshold, the step A02 is re-executed. If the current number of iterations is not less than the preset maximum number of iterations threshold, the classification is completed. That is, step A02 is re-executed until the cluster center after the update is the same as the cluster center before the update, or the current number of iterations is equal to the preset maximum number of iterations, then the classification of the log vector is completed.

[0093] See also Figure 3 As shown in FIG. 1 , a log processing method is shown. First, a log to be classified is input, and relevant classification parameters are initialized, and the preset maximum number of iterations is reduced by 1. Then, a log to be classified other than the cluster center in the log to be classified is taken as an ant, and the state transition probability of ant i is calculated, that is, the probability of dividing the ant into the class corresponding to each cluster center is calculated, and the ant is divided into the class (cluster) corresponding to the corresponding cluster center according to this probability, and it is judged whether each ant is divided into the corresponding cluster. If so, the cluster center is recalculated, and the pheromone of each ant to the cluster center is updated, and it is judged whether the termination condition is met. If it is met, the final solution is output, and if it is not met, the steps of calculating the state transition probability of ant i and dividing the ant into the class corresponding to the corresponding cluster center according to this probability are repeated.

[0094] See also Figure 4 As shown, the embodiment of the present application discloses a log processing device, including:

[0095] The data acquisition module 11 is used to acquire the logs to be classified;

[0096] The feature item extraction module 12 is used to extract feature items from each log in the log to be classified, and obtain a set of log feature items corresponding to each log in the log to be classified;

[0097] A log vector determination module 13, configured to determine a log vector corresponding to each log in the log to be classified based on a set of log feature items corresponding to each log in the log to be classified;

[0098] The log classification module 14 is used to classify the log vectors corresponding to each log in the log to be classified by using an ant colony clustering algorithm, so as to classify the log to be classified.

[0099] It can be seen that the present application first obtains the log to be classified, and extracts the feature items of each log in the log to be classified, obtains the log feature item set corresponding to each log in the log to be classified, and then determines the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified, and then the ant colony clustering algorithm can be used to classify the log vector corresponding to each log in the log to be classified, so as to classify the log to be classified. In this way, after obtaining the log to be classified, feature extraction and other processing are performed on the obtained log to be classified to obtain the corresponding log vector, and then the ant colony clustering algorithm is used to classify each log vector, so that the obtained log to be classified can be classified. Compared with the prior art that requires the input of prior information, the ant colony clustering algorithm does not require the setting of prior information, thereby reducing the burden on users, improving clustering results, and improving applicability. The essence of the ant colony clustering algorithm is a very effective random search mechanism, rather than a heuristic mechanism, thereby improving the accuracy and consistency of the classification results. In addition, the algorithm structure and operation of the ant colony clustering algorithm are relatively simple and easy to implement.

[0100] Specifically, the data acquisition module 11 is also used for:

[0101] Obtain the log classification parameters corresponding to the log to be classified.

[0102] Furthermore, the log classification module 14 is used to:

[0103] A01: determining an initial cluster center from the log vector, and determining pheromones from the log vectors to be classified except the initial cluster center in the log vector to each of the initial cluster centers;

[0104] A02: dividing each of the log vectors to be classified into a class corresponding to the initial cluster center based on the pheromone;

[0105] A03: Determine the sum of distances from each log vector to other log vectors except itself;

[0106] A04: updating the initial cluster center based on the distance and the pheromone;

[0107] Step A02 is executed again until the cluster center after updating is the same as the cluster center before updating, or the current number of iterations is equal to the preset maximum number of iterations, and the classification of the log vector is completed.

[0108] Furthermore, the log classification module 14 is used to:

[0109] Based on the first operation formula, the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers is determined, wherein the first operation formula is:

[0110]

[0111] Among them, τ ij represents the pheromone from the i-th log vector to be classified to the j-th initial cluster center, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th initial cluster center, and r represents the preset cluster center radius.

[0112] Furthermore, the log classification module 14 is used to:

[0113] Determine the probability of classifying each of the to-be-classified log vectors into the class corresponding to each of the initial cluster centers based on the pheromone and the second operation formula;

[0114] According to the probability, each of the log vectors to be classified is divided into the class corresponding to the initial cluster center, wherein the second operation formula is:

[0115]

[0116] Among them, the P ij It represents the probability of classifying the i-th log vector to be classified into the class corresponding to the j-th initial clustering center. α and β are both preset adjustment factors. S represents the set of log vectors to be classified whose Euclidean distance to the j-th initial clustering center is less than or equal to the preset clustering center radius.

[0117] Furthermore, the log classification module 14 is used to:

[0118] The sum of the distances of each log vector to other log vectors other than itself is determined based on a third operation formula, wherein the third operation formula is:

[0119]

[0120]

[0121] Among them, L m represents the sum of the distances from the mth log vector to other log vectors except itself, x m represents the mth log vector, x mp is the pth value of the mth log vector, N represents the total number of log vectors, c mp is the transition vector c m The pth value in ||x m -cm || 2 represents the log vector x m and the transition vector c m The square of the modulus.

[0122] Furthermore, the log classification module 14 is used to:

[0123] Determine the minimum distance and the corresponding log vector as a new cluster center, and use the new cluster center to update the initial cluster center;

[0124] The pheromone is updated using a fourth operation formula, wherein the fourth operation formula is:

[0125]

[0126] Among them, τ ij ′ represents the pheromone of the i-th log vector to be classified to the j-th cluster center after updating, τ ij represents the pheromone from the i-th log vector to be classified to the j-th cluster center before updating, d ij represents the Euclidean distance from the i-th log vector to be classified to the j-th cluster center, ρ represents the volatility of the pheromone, and Q represents the total amount of preset pheromone.

[0127] See also Figure 5 , which is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include but is not limited to a laptop computer, a desktop computer or a server.

[0128] Generally, the electronic device 20 in this embodiment includes: a processor 21 and a memory 22 .

[0129] Among them, the processor 21 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 21 may be implemented using at least one of DSP (digital signal processing), FPGA (field-programmable gate array), and PLA (programmable logic array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (central processing unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a GPU (graphics processing unit), which is responsible for rendering and drawing images to be displayed on the display screen. In some embodiments, the processor 21 may include an AI (artificial intelligence) processor, which is used to process computing operations related to machine learning.

[0130] The memory 22 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 22 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 22 is at least used to store the following computer program 221, wherein after the computer program is loaded and executed by the processor 21, the steps of the log processing method disclosed in any of the aforementioned embodiments can be implemented.

[0131] In some embodiments, the electronic device 20 may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a sensor 26 , a power source 27 , and a communication bus 28 .

[0132] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device 20, and may include more or fewer components than shown in the figure.

[0133] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the log processing method disclosed in any of the aforementioned embodiments.

[0134] For the specific process of the log processing method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0137] Finally, it should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a series of processes, methods, articles or equipment containing other elements include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0138] The above is a detailed introduction to a log processing method, device, equipment, and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A log processing method, characterized in that: include: Get the logs to be classified; Extracting feature items from each log in the logs to be classified to obtain a set of log feature items corresponding to each log in the logs to be classified; Determine the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified; Using an ant colony clustering algorithm to classify the log vectors corresponding to each log in the log to be classified, so as to classify the log to be classified; The step of using an ant colony clustering algorithm to classify the log vectors corresponding to each log in the log to be classified includes: A01: determining an initial cluster center from the log vector, and determining pheromones from the log vectors to be classified except the initial cluster center in the log vector to each of the initial cluster centers; A02: dividing each of the log vectors to be classified into a class corresponding to the initial cluster center based on the pheromone; A03: Determine the sum of distances from each log vector to other log vectors except itself; A04: updating the initial cluster center based on the distance and the pheromone; Re-execute step A02 until the cluster center after the update is the same as the cluster center before the update, or the current number of iterations is equal to the preset maximum number of iterations, then the classification of the log vector is completed; The step of determining the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers includes: Based on the first operation formula, the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers is determined, wherein the first operation formula is: ； in, Indicates log vectors to be classified The pheromone of the initial cluster center, Indicates log vectors to be classified The Euclidean distance of the initial cluster centers, Indicates the preset cluster center radius; The dividing each of the to-be-classified log vectors into the class corresponding to the initial cluster center based on the pheromone includes: Determine the probability of classifying each of the to-be-classified log vectors into the class corresponding to each of the initial cluster centers based on the pheromone and the second operation formula; According to the probability, each of the log vectors to be classified is divided into the class corresponding to the initial cluster center, wherein the second operation formula is: ； Among them, the Indicates that the The log vectors to be classified are divided into The probability of being in the class corresponding to the initial cluster center, , All are preset adjustment factors. Indicates to a set of log vectors to be classified whose Euclidean distance to the initial cluster centers is less than or equal to the preset cluster center radius; Determining the sum of distances of each log vector to other log vectors other than itself includes: The sum of the distances of each log vector to other log vectors other than itself is determined based on a third operation formula, wherein the third operation formula is: ； ； in, Indicates The sum of the distances of a log vector to other log vectors except itself, Indicates log vector, For the The log vector values, represents the total number of log vectors, is the transition vector The values, Represents a log vector and transition vector The square of the modulus; The updating of the initial cluster center based on the distance and the updating of the pheromone includes: Determine the minimum distance and the corresponding log vector as a new cluster center, and use the new cluster center to update the initial cluster center; The pheromone is updated using a fourth operation formula, wherein the fourth operation formula is: ； in, Indicates the updated log vectors to be classified The pheromone of the cluster center, Indicates the update before log vectors to be classified The pheromone of the cluster center, Indicates log vectors to be classified The Euclidean distance of the cluster centers, Indicates the volatility of pheromone, Indicates the preset total amount of pheromones.

2. The log processing method according to claim 1, characterized in that: After obtaining the logs to be classified, the method further includes: Obtain the log classification parameters corresponding to the log to be classified.

3. A log processing device, characterized in that: include: A data acquisition module is used to obtain logs to be classified; A feature item extraction module is used to extract feature items from each log in the log to be classified, and obtain a set of log feature items corresponding to each log in the log to be classified; A log vector determination module, used to determine the log vector corresponding to each log in the log to be classified based on the log feature item set corresponding to each log in the log to be classified; A log classification module, used to classify the log vectors corresponding to each log in the log to be classified by using an ant colony clustering algorithm, so as to classify the log to be classified; The step of using an ant colony clustering algorithm to classify the log vectors corresponding to each log in the log to be classified includes: A01: determining an initial cluster center from the log vector, and determining pheromones from the log vectors to be classified except the initial cluster center in the log vector to each of the initial cluster centers; A02: dividing each of the log vectors to be classified into a class corresponding to the initial cluster center based on the pheromone; A03: Determine the sum of distances from each log vector to other log vectors except itself; A04: updating the initial cluster center based on the distance and the pheromone; Re-execute step A02 until the cluster center after updating is the same as the cluster center before updating, or the current number of iterations is equal to the preset maximum number of iterations, and the classification of the log vector is completed; The step of determining the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers includes: Based on the first operation formula, the pheromone of the log vector to be classified except the initial cluster center in the log vector to each of the initial cluster centers is determined, wherein the first operation formula is: ； in, Indicates log vectors to be classified The pheromone of the initial cluster center, Indicates log vectors to be classified The Euclidean distance of the initial cluster centers, Indicates the preset cluster center radius; The dividing each of the to-be-classified log vectors into the class corresponding to the initial cluster center based on the pheromone includes: Determine the probability of classifying each of the to-be-classified log vectors into the class corresponding to each of the initial cluster centers based on the pheromone and the second operation formula; According to the probability, each of the log vectors to be classified is divided into the class corresponding to the initial cluster center, wherein the second operation formula is: ； Among them, the Indicates that the The log vectors to be classified are divided into The probability of being in the class corresponding to the initial cluster center, , All are preset adjustment factors. Indicates to a set of log vectors to be classified whose Euclidean distance to the initial cluster centers is less than or equal to the preset cluster center radius; Determining the sum of distances of each log vector to other log vectors other than itself includes: The sum of the distances of each log vector to other log vectors other than itself is determined based on a third operation formula, wherein the third operation formula is: ； ； in, Indicates The sum of the distances of a log vector to other log vectors except itself, Indicates log vector, For the The log vector values, represents the total number of log vectors, is the transition vector The values, Represents a log vector and transition vector The square of the modulus; The updating of the initial cluster center based on the distance and the updating of the pheromone includes: Determine the minimum distance and the corresponding log vector as a new cluster center, and use the new cluster center to update the initial cluster center; The pheromone is updated using a fourth operation formula, wherein the fourth operation formula is: ； in, Indicates the updated log vectors to be classified The pheromone of the cluster center, Indicates the update before log vectors to be classified The pheromone of the cluster center, Indicates log vectors to be classified The Euclidean distance of the cluster centers, Indicates the volatility of pheromone, Indicates the preset total amount of pheromones.

4. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store computer programs; The processor is used to execute the computer program to implement the log processing method according to claim 1 or 2.

5. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program, when executed by a processor, implements the log processing method according to claim 1 or 2.

Images