Log processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a log processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-processed log, and extracting a target segmented word in the to-be-processed log to obtain a to-be-processed segmented word group; the to-be-processed segmented word group is converted through a preset log template tree, a target log conforming to the preset log template tree is obtained, and the preset log template tree is obtained after word segmentation screening processing and arrangement are conducted on preset logs based on a preset word database; preset segmented words and word vectors corresponding to the preset segmented words are stored in the preset word database, and the word vectors are determined based on the positions of the preset segmented words in the log. The method can simplify the process and improve the efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Typically, log files contain various information such as system activities, error messages, event records, etc., but log files can be stored in a variety of ways and are difficult to analyze. Therefore, by analyzing the common patterns and keywords of a large amount of log data, important information can be automatically identified and captured, and converted into a structured format for subsequent data mining, visualization, or alert generation.

[0003] In related technologies, a large amount of log data can be analyzed through clustering algorithms, etc., and the similarities between log entries can be measured to group log information and classify them. However, this method requires a large amount of data to be processed and is computationally difficult, making it difficult to efficiently cope with large-scale log event streams. Summary of the invention

[0004] Based on this, it is necessary to provide a log processing method, device, computer equipment, computer-readable storage medium and computer program product that can simplify the process and improve efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a log processing method, comprising:

[0006] Obtaining a log to be processed, and extracting target segmented words in the log to be processed to obtain a segmented word group to be processed;

[0007] The word segmentation group to be processed is converted using a preset log template tree to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arrangement of the preset log based on a preset word database, wherein the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segmentation in the log.

[0008] In one embodiment, the method of acquiring the preset word database includes:

[0009] Obtain an initial log, and divide the initial log to obtain a plurality of initial participles, each of the initial participles corresponding to a word frequency;

[0010] Determine the initial segmented words whose word frequency is greater than the preset word frequency as the preset segmented words;

[0011] Determining the association relationship between the preset participles according to the positions of the preset participles in the log;

[0012] According to the association relationship, the word vector corresponding to each of the preset word segments is determined to generate a preset word database.

[0013] In one embodiment, the method for determining the preset log template tree includes:

[0014] Obtaining a preset log, and dividing the preset log to obtain multiple candidate segmentations;

[0015] Determine the candidate segmented words that match the preset word database as matching segmented words, and construct a word chain list according to the position and occurrence frequency of the matching segmented words in the preset log;

[0016] An initial log tree is generated according to the word chain list, and the nodes of the initial log tree are adjusted to obtain a preset log template tree.

[0017] In one embodiment, after adjusting the nodes of the initial log tree, a preset log template tree is obtained, including:

[0018] Determine the number of child nodes corresponding to each node in the initial log tree;

[0019] Determine a node whose number of child nodes is greater than or equal to a first preset node threshold as a deletion node, and delete the child nodes corresponding to the deletion node to obtain a candidate log tree;

[0020] Determine a node in the candidate log tree whose number of child nodes is greater than a second preset node threshold and less than a first preset node threshold as a node to be pruned;

[0021] After processing the child nodes of the node to be pruned using the preset pruning strategy, a preset log template tree is obtained.

[0022] In one embodiment, the processing of the child nodes of the node to be pruned by using a preset pruning strategy includes:

[0023] For each node to be pruned, determining a child node corresponding to the node to be pruned as a brother node;

[0024] Determine the similarity between the brother nodes according to the word vectors corresponding to the brother nodes;

[0025] When the similarity is greater than a preset threshold, a node whose similarity to the node to be pruned is less than a preset value is determined from the sibling nodes as a pruned node, and the pruned node is deleted to obtain a processed node to be pruned.

[0026] In one embodiment, after adjusting the nodes of the initial log tree, the process further includes:

[0027] For the adjusted initial log tree, set the pointer of the root node to be empty and the pointers of the direct child nodes of the root node to point to the root node;

[0028] For each node, search from the pointer of the parent node of the node to the leaf nodes step by step until the target node is determined, and set the pointer of the node to point to the target node, wherein the target node has the same child node as the node.

[0029] In a second aspect, the present application also provides a log processing device, including:

[0030] An acquisition module is used to acquire the log to be processed, and extract the target word segmentation in the log to be processed to obtain the word segmentation group to be processed;

[0031] A conversion module is used to convert the word group to be processed using a preset log template tree to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arranging the preset log based on a preset word database, and the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segment in the log.

[0032] In one embodiment, the device further includes a module for acquiring the preset word database, and the module for acquiring the preset word database is further used for:

[0033] Obtain an initial log, and divide the initial log to obtain a plurality of initial participles, each of the initial participles corresponding to a word frequency;

[0034] Determine the initial segmented words whose word frequency is greater than the preset word frequency as the preset segmented words;

[0035] Determining the association relationship between the preset participles according to the positions of the preset participles in the log;

[0036] According to the association relationship, the word vector corresponding to each of the preset word segments is determined to generate a preset word database.

[0037] In one of the embodiments, the device further includes a module for determining the preset log template tree, and the module for determining the preset log template tree is further used to:

[0038] Obtaining a preset log, and dividing the preset log to obtain multiple candidate segmentations;

[0039] Determine the candidate segmented words that match the preset word database as matching segmented words, and construct a word chain list according to the position and occurrence frequency of the matching segmented words in the preset log;

[0040] An initial log tree is generated according to the word chain list, and the nodes of the initial log tree are adjusted to obtain a preset log template tree.

[0041] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0042] Determine the number of child nodes corresponding to each node in the initial log tree;

[0043] Determine a node whose number of child nodes is greater than or equal to a first preset node threshold as a deletion node, and delete the child nodes corresponding to the deletion node to obtain a candidate log tree;

[0044] Determine a node in the candidate log tree whose number of child nodes is greater than a second preset node threshold and less than a first preset node threshold as a node to be pruned;

[0045] After processing the child nodes of the node to be pruned using the preset pruning strategy, a preset log template tree is obtained.

[0046] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0047] For each node to be pruned, determining a child node corresponding to the node to be pruned as a brother node;

[0048] Determine the similarity between the brother nodes according to the word vectors corresponding to the brother nodes;

[0049] When the similarity is greater than a preset threshold, a node whose similarity to the node to be pruned is less than a preset value is determined from the sibling nodes as a pruned node, and the pruned node is deleted to obtain a processed node to be pruned.

[0050] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0051] For the adjusted initial log tree, set the pointer of the root node to be empty and the pointers of the direct child nodes of the root node to point to the root node;

[0052] For each node, search from the pointer of the parent node of the node to the leaf nodes step by step until the target node is determined, and set the pointer of the node to point to the target node, wherein the target node has the same child node as the node.

[0053] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0054] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0055] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0056] The above-mentioned log processing method, device, computer equipment, computer-readable storage medium and computer program product, when processing logs, obtain the log to be processed, extract the target word segmentation in the log to be processed, obtain the word segmentation group to be processed, use the preset log template tree to convert the word segmentation group to be processed, and obtain the target log that conforms to the preset log template tree, so that the log to be processed can be converted into a structured format according to the preset log template tree to obtain the target log, and automatically identify and capture important information through the preset log template tree, which is convenient for subsequent data mining, visualization, alarm generation, etc., and can more effectively analyze and manage the logs. When creating a preset log template tree, the preset log is segmented, filtered, processed and arranged based on the preset word database. The preset word database stores preset segmentations and corresponding word vectors. The word vector is determined based on the position of the preset segmentation in the log. Therefore, by setting the corresponding relationship between the word vector and the segmentation, the deep association between the segmentations in the log can be captured, which improves the accuracy and reliability of the generated preset log template tree, facilitates the subsequent optimization processing such as pruning and merging of the log template tree, optimizes the construction of the template tree, improves the intelligence, real-time and efficiency of log data processing in a big data environment, simplifies the log processing process, improves the log processing efficiency, and is suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A flow chart of a log processing method in one embodiment;

[0059] Figure 2 Schematic diagram of a flow chart of a method for determining a word vector in one embodiment;

[0060] Figure 3Schematic diagram of candidate word segmentations in one embodiment;

[0061] Figure 4 A flow chart of a log processing method according to another embodiment;

[0062] Figure 5 A flow chart of a log processing method according to another embodiment;

[0063] Figure 6 is a structural block diagram of a log processing device in an embodiment;

[0064] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In one embodiment, Figure 1 As shown, a log processing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0067] Step S110, obtaining a log to be processed, and extracting target segmented words in the log to be processed to obtain a segmented word group to be processed;

[0068] Exemplarily, the log to be processed is obtained, and the target word segmentation in the log to be processed is extracted to obtain the word segmentation group to be processed. The log to be processed can be collected in real time, and when new log data is generated, the new log data is obtained to obtain the log to be processed.

[0069] In one example, when extracting a target word segmentation from a log to be processed, the log to be processed can be first divided to obtain multiple original word segmentations, and then the original word segmentations can be screened to obtain the target word segmentation, wherein the screening can be based on a preset word database, such as screening the word segmentation in the preset word database from multiple original word segmentations as the target word segmentation. In one example, before extracting the target word segmentation from a log to be processed, the log to be processed can be preprocessed, and the preprocessing process can include but is not limited to noise reduction, deduplication, etc., which can be determined based on the actual application scenario.

[0070] In a possible implementation, the log to be processed may be directly divided to obtain target word segments.

[0071] Optionally, the target segmented words in the log to be processed constitute a segmented word group to be processed. One segmented word group to be processed may include one or more target segmented words, which can be specifically determined according to the log to be processed and the log division method in an actual application scenario.

[0072] Step S120, using a preset log template tree to convert the word group to be processed to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arrangement of the preset log based on a preset word database, and the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segment in the log.

[0073] Exemplarily, the word group to be processed is converted using a preset log template tree to obtain a target log that conforms to the preset log template tree. In one example, each word in the word group to be processed corresponds to a word frequency, and a query can be performed in the preset log template tree starting from the word with the highest word frequency to determine a node route that matches the word group to be processed, and the log corresponding to the node route is the target log. In one example, each node in the preset log template tree corresponds to a word.

[0074] Optionally, the preset log template tree is obtained by performing word segmentation, screening, processing and arranging of preset logs based on a preset word database. In one example, the preset log can be determined based on historical logs, for example, the preset log can be a log generated within a preset historical period, the preset log can include multiple logs, and the number of preset logs can be determined based on actual application scenarios.

[0075] In one example, the preset word database is a database storing preset word segments and word vectors corresponding to the preset word segments, wherein each preset word segment corresponds to a word vector, and different preset word segmentations may correspond to different or the same word vectors, which may be determined specifically according to the actual application scenario. The word vector is determined based on the position of the preset word segment in the log. In one example, the word vector used to characterize the context association relationship of the word segment can be determined based on the position of the preset word segment in the log and the association relationship with the context, wherein the word vectors of word segments with similar contexts are closer. The word vector maps each word to a real number vector, so that each word has a unique representation in a multidimensional vector space.

[0076] In a possible implementation, a word vector model can be constructed using algorithms such as Word2Vec and BERT, and the word vector model can be used to determine the word vector of the preset word segmentation. When constructing a preset log template tree, the template tree can be pruned or node adjusted according to the word vector corresponding to each node, which can be determined according to the actual application scenario.

[0077] In the disclosed embodiment, when processing logs, a log to be processed is obtained, target word segments in the log to be processed are extracted, and word segments groups to be processed are obtained. The word segments groups to be processed are converted using a preset log template tree to obtain a target log that conforms to the preset log template tree. Thus, the log to be processed can be converted into a structured format according to the preset log template tree to obtain a target log. Important information is automatically identified and captured through the preset log template tree, which facilitates subsequent data mining, visualization, alarm generation, etc., and can more effectively analyze and manage logs. When creating a preset log template tree, the preset log is segmented, filtered, processed and arranged based on the preset word database. The preset word database stores preset segmentations and corresponding word vectors. The word vector is determined based on the position of the preset segmentation in the log. Therefore, by setting the corresponding relationship between the word vector and the segmentation, the deep association between the segmentations in the log can be captured, which improves the accuracy and reliability of the generated preset log template tree, facilitates the subsequent optimization processing such as pruning and merging of the log template tree, optimizes the construction of the template tree, improves the intelligence, real-time and efficiency of log data processing in a big data environment, simplifies the log processing process, improves the log processing efficiency, and is suitable for more application scenarios.

[0078] In one embodiment, the method of acquiring the preset word database includes:

[0079] Obtain an initial log, and divide the initial log to obtain a plurality of initial participles, each of the initial participles corresponding to a word frequency;

[0080] Determine the initial segmented words whose word frequency is greater than the preset word frequency as the preset segmented words;

[0081] Determining the association relationship between the preset participles according to the positions of the preset participles in the log;

[0082] According to the association relationship, the word vector corresponding to each of the preset word segments is determined to generate a preset word database.

[0083] Exemplarily, the preset word database is constructed using the initial log, wherein the initial log may be the same as the preset log or may be different from the preset log. For example, the initial log may be obtained by filtering the historical logs.

[0084] In one example, the initial log is divided to obtain multiple initial participles, each initial participle corresponds to a word frequency, and the initial participle with a word frequency greater than a preset word frequency is determined as the preset participle. The word frequency can be the frequency of occurrence of the initial participle in the initial log. Generally, the higher the word frequency, the higher the importance of the participle. In one example, when the initial log is divided, the obtained participles can also be filtered to obtain the initial participles. For example, the participles of entity attributes or operation attributes are filtered as initial participles according to the participle attributes. The preset word frequency can be set according to the actual application scenario. When the word frequency is greater than the preset word frequency, the participle can be considered to be of high importance and determined as the preset participle.

[0085] According to the position of the preset segmentation in the log, the association relationship between the preset segmentation is determined, for example, the previous adjacent segmentation and the next adjacent segmentation of the preset segmentation. According to the association relationship, the word vector corresponding to each preset segmentation is determined. For example, the higher the similarity between the previous adjacent segmentation and the next adjacent segmentation of two preset segmentations, the closer the word vectors of the two preset segmentations are. In one example, the word vector model can be trained to learn the contextual relationship of the segmentation, and the word vector is output so that the segmentations with similar contexts are closer in the vector space.

[0086] Figure 2 A method for determining a word vector according to an exemplary embodiment is shown. The original log data is obtained, and after data preprocessing such as denoising and sentence segmentation, a word vector model is trained. The word vector model is used to output word segmentation and corresponding word vectors to form a vocabulary, that is, a preset word database.

[0087] In the disclosed embodiment, a preset word database is determined according to an initial log, the initial log is divided to obtain a plurality of initial participles, initial participles with higher word frequencies are determined as preset participles, and the association relationship between the preset participles is determined according to the positions of the preset participles in the log, and a word vector corresponding to each preset participle is determined according to the association relationship, and a preset word database is generated, so that the preset participles with higher word frequencies can be screened out using the initial log, and word vectors for characterizing the contextual relationships of the preset participles are obtained, thereby ensuring the accuracy and reliability of the preset participles in the preset word database. At the same time, the word vectors are used to facilitate a deep understanding and capture of the deep associations between the participles within the log entry.

[0088] In one embodiment, the method for determining the preset log template tree includes:

[0089] Obtaining a preset log, and dividing the preset log to obtain multiple candidate segmentations;

[0090] Determine the candidate segmented words that match the preset word database as matching segmented words, and construct a word chain list according to the position and occurrence frequency of the matching segmented words in the preset log;

[0091] An initial log tree is generated according to the word chain list, and the nodes of the initial log tree are adjusted to obtain a preset log template tree.

[0092] Exemplarily, when determining the preset log template tree, a preset log is obtained, and the preset log is divided to obtain multiple candidate segmentations. In one example, the division method of the preset log is similar to the division method of the initial log in the above embodiment.

[0093] In one example, among multiple candidate participles, candidate participles that match the preset word database are screened as matching participles. For example, when the same participle can be found in the preset word database, the candidate participle can be determined to be a matching participle.

[0094] Figure 3 This is a schematic diagram of a candidate word segmentation according to an exemplary embodiment. The original log data is preprocessed and segmented, and sorted according to word frequency to obtain a log word segmentation sequence. According to the log word segmentation sequence, a word chain list can be generated.

[0095] Optionally, a word chain list is constructed according to the position and frequency of occurrence of the matching word in the preset log. In one example, the matching word can be sorted according to the frequency of occurrence in the preset log to construct a word chain list, each chain list starts with the most frequent word, and the remaining words are arranged in descending order of word frequency. In one example, each word has a pointer to a word vector.

[0096] Exemplarily, according to the common prefix principle, the word linked lists are merged to obtain an initial log tree of a multi-branch tree structure. After adjusting the nodes of the initial log tree, a preset log template tree is obtained, wherein the adjustment process may include but is not limited to node pruning, node merging, node pointer setting, etc., which can be determined according to the actual application scenario.

[0097] In the disclosed embodiment, when generating a preset log template tree, the preset log is first divided, and matching segmented words are screened as matching segmented words, a word chain list is constructed according to the position and occurrence frequency of the matching segmented words in the preset log, and the preset log template tree is obtained according to the word quantity table, so that the preset log template tree that can characterize the context association relationship of the segmented words and the occurrence frequency of the segmented words can be quickly and accurately generated, so that the preset log template tree can accurately reflect the association relationship between the segmented words in the log and the importance of the segmented words, thereby improving the accuracy and reliability of the target log obtained after log processing.

[0098] In one embodiment, after adjusting the nodes of the initial log tree, a preset log template tree is obtained, including:

[0099] Determine the number of child nodes corresponding to each node in the initial log tree;

[0100] Determine a node whose number of child nodes is greater than or equal to a first preset node threshold as a deletion node, and delete the child nodes corresponding to the deletion node to obtain a candidate log tree;

[0101] Determine a node in the candidate log tree whose number of child nodes is greater than a second preset node threshold and less than a first preset node threshold as a node to be pruned;

[0102] After processing the child nodes of the node to be pruned using the preset pruning strategy, a preset log template tree is obtained.

[0103] Exemplarily, when adjusting the nodes of the initial log tree, pruning processing may be performed according to the number of child nodes corresponding to each node.

[0104] In one example, the number of child nodes corresponding to each node is determined, and a first preset node threshold m and a second preset node threshold n are set, where m>n. The first preset node threshold is an upper limit of the number of nodes set according to an actual application scenario, and the second preset node threshold is a lower limit of the number of nodes set according to an actual application scenario. When the number of child nodes is greater than or equal to the first preset node threshold, it can be considered that the number of child nodes of the node is too large at this time, and the node whose number of child nodes is greater than or equal to the first preset node threshold is determined as a deleted node, and the child nodes corresponding to the node are deleted to obtain a candidate log tree.

[0105] Optionally, for a node in the candidate log tree whose number of child nodes is between the second preset node threshold and the first preset node threshold, it can be determined as a node to be pruned, and the child nodes of the node to be pruned are processed using a preset pruning strategy to obtain a preset log template tree. The preset pruning strategy can be determined according to an actual application scenario, for example, partial pruning can be performed according to a word vector, partial pruning can be performed according to a relationship between a node and other nodes, etc.

[0106] In an example, when the number of child nodes is less than the second preset node threshold, it can be considered that the number of child nodes of the node is small, and no pruning process is performed and the node is directly retained.

[0107] The disclosed embodiment prunes the log tree according to the number of child nodes of each node. Different numbers of child nodes correspond to different pruning strategies. Nodes with too many child nodes are deleted, thereby ensuring the simplicity and reliability of the preset log template tree finally obtained, effectively reducing the data processing workload in the subsequent log processing process, improving the efficiency of processing the logs to be processed using the preset log template tree, and being suitable for more application scenarios.

[0108] In one embodiment, the processing of the child nodes of the node to be pruned by using a preset pruning strategy includes:

[0109] For each node to be pruned, determining a child node corresponding to the node to be pruned as a brother node;

[0110] Determine the similarity between the brother nodes according to the word vectors corresponding to the brother nodes;

[0111] When the similarity is greater than a preset threshold, a node whose similarity to the node to be pruned is less than a preset value is determined from the sibling nodes as a pruned node, and the pruned node is deleted to obtain a processed node to be pruned.

[0112] Exemplarily, the word vector of the node can be used to prune the node to be pruned. For each node to be pruned, the child node corresponding to the node to be pruned is determined as a sibling node, and the similarity between the sibling nodes is determined according to the word vector corresponding to the sibling node. For example, the similarity between the sibling nodes can be obtained by calculating the cosine similarity of the word vectors between the nodes.

[0113] In one example, the similarities between the sibling nodes may be calculated to obtain multiple similarities.

[0114] When the similarity is greater than a preset threshold, it can be considered that the semantic similarity of the corresponding nodes is high. In order to ensure the simplicity and compactness of the template tree, some nodes can be deleted. At this time, the nodes with less similarity to the parent node (i.e., the node to be pruned) can be determined as pruned nodes, and the pruned nodes can be deleted to obtain the processed nodes to be pruned. In an example, the preset threshold can be set according to the actual application scenario. When the similarity is greater than the preset threshold, it can be considered that the semantic similarity between the two is high.

[0115] Optionally, the preset value can be determined according to the actual application scenario, and different sibling nodes can correspond to different preset values. For example, the higher value of the similarity between the sibling node and the node to be pruned can be determined as the preset value, and the pruned node is the sibling node with a lower similarity to the node to be pruned.

[0116] In one example, if the similarity is less than or equal to a preset threshold, it means that the semantic similarity between the sibling nodes is close, and the node can be directly retained without pruning.

[0117] The disclosed embodiment determines the similarity through the word vectors of the child nodes of the node to be pruned, and deletes and prunes the nodes with higher similarity, thereby realizing refined pruning. The similarity is calculated through the word vectors, which effectively improves the accuracy and efficiency of the pruning calculation, and quickly and accurately selects the nodes with higher similarity for pruning, taking into account the accuracy and simplicity of the preset log template tree, effectively filtering out irrelevant log templates, and improving the compactness and flexibility of the template tree.

[0118] In one embodiment, after adjusting the nodes of the initial log tree, the method further includes:

[0119] For the adjusted initial log tree, set the pointer of the root node to be empty and the pointers of the direct child nodes of the root node to point to the root node;

[0120] For each node, search from the pointer of the parent node of the node to the leaf nodes step by step until the target node is determined, and set the pointer of the node to point to the target node, wherein the target node has the same child node as the node.

[0121] Exemplarily, in order to speed up the log processing process, a pointer pointing to another node is also configured for the node in the template tree. In one example, the pointer of the root node is first set to null, and the pointer of the direct child node of the root node is set to point to the root node; then, each node is traversed from the root node in turn, and for each node, the node pointed to by the pointer of the parent node of the node is searched step by step to the leaf node until a target node is found, so that the word corresponding to the target node can be found from the child nodes of the target node, that is, the target node has the same child node as the node, and the pointer of the node is set to point to the target node.

[0122] In one example, if the pointer of the parent node of the node is null, the pointer of the node is also null. Exemplarily, when setting the pointer, you can start from the root node and put the root node into the queue. Take a node from the queue and traverse all its child nodes. For each child node, set the pointer of the child node according to the pointer of the current node. If the pointer is set, add the child node to the queue and continue processing. Until the queue is empty, the construction is completed. Optionally, each node pointer points to a node in the longest suffix of the node (excluding the current node) that also exists in the log tree.

[0123] The disclosed embodiment configures a pointer for each node in the template tree, thereby significantly accelerating the query speed during log processing and improving the log processing efficiency when processing long string matches, and is suitable for more application scenarios.

[0124] In an example, the template tree may also be updated and optimized according to the new online log, such as Figure 4 As shown, after obtaining a new online log, preprocessing is performed to obtain a word segmentation sequence, and the word segmentation sequence is used to update a preset log template tree, and the updated preset log template tree is used to extract the template corresponding to the online log.

[0125] Figure 5 FIG. 1 is a flow chart of a log processing method according to an exemplary embodiment. Figure 5As shown, the word vector model training stage is used to construct a word vector model that can accurately map the semantic associations between log keywords. Preprocessing and vocabulary construction: First, extract log data from the log library and perform denoising and sentence splitting operations. Construct a vocabulary from the preprocessed logs and remove low-frequency words. Model training: Use word vector models such as Word2Vec and BERT to learn the context relationships of log word segmentation, so that words with similar contexts are closer in the vector space. Add word vector attributes to the word segmentation of the vocabulary.

[0126] Construction of the basic word frequency tree: Traverse the preprocessed logs, sort the word segmentation results by word frequency, and construct a log vocabulary sequence. Subsequently, create a vocabulary linked list based on the sorting results. Each linked list starts with the highest-frequency word, and the remaining words are arranged in descending order and have pointers to word vectors. Finally, merge the linked lists into a multi-way tree structure according to the common prefix principle.

[0127] Optimization pruning strategy: Traverse and prune the initially constructed template tree. The optimization pruning strategy is as follows: Sub-node number threshold rule: Define the upper and lower limits m and n of the sub-node number threshold. Count the number of sub-nodes x of each node. If x >= m, delete all the sub-nodes of this node to make this node a leaf node. If x <= n, no pruning is performed. If n < x < m, the following similarity threshold condition pruning is performed. Sibling node word vector similarity threshold rule: Define the sibling node similarity threshold s. Calculate the word vector similarity between sibling nodes, extract the word vector attributes of the nodes, and calculate the cosine similarity t of the word vectors between the nodes. If the similarity t between sibling nodes > s, greater than the threshold, it indicates a high semantic similarity. Retain the node with a high similarity to the parent node and delete other sub-nodes for pruning. If the similarity t between sibling nodes <= s, less than the threshold, it indicates a low semantic similarity, and retain the nodes.

[0128] The node integrates the fail pointer. In order to speed up the template matching process, the node of the template tree is configured with a fail pointer pointing to another node. The fail pointer points to the node in the longest suffix of the current node (excluding the current node) that also exists in the tree. The construction rules are as follows: the fail pointer of the root node is empty; the fail pointer of the direct child node of the root node points to the root node. Other nodes: set according to the fail pointer of its parent node. If the fail pointer of the parent node is empty, the fail pointer of the node is also empty. Otherwise, starting from the fail pointer of the parent node, search upward step by step until a node is found so that the word represented by the current node can be found from the child nodes of the node. Point the fail pointer of the current node to the found node. Construction steps: Start from the root node and put the root node into the queue. Take a node from the queue and traverse all its child nodes. For each child node: set the fail pointer of the child node according to the fail pointer of the current node. If the fail pointer is set, add the child node to the queue and continue processing. The construction ends when the queue is empty.

[0129] Node merging strategy, merge path nodes with only a single child node to achieve space optimization. In the basic template tree, some nodes have only a single child node, and these nodes will also be retained, resulting in a waste of storage space. These parent nodes with only a single child node are merged with their child nodes to reduce the depth and breadth of the tree, improve space utilization and query efficiency.

[0130] Online log processing mechanism, the above method is suitable for real-time online environment, can timely parse and analyze the continuously generated log data, and can dynamically update the template tree to adapt to the changing log data. When new log data is generated, the system can quickly perform preprocessing, word segmentation, word frequency sorting and other operations, and immediately integrate these new data into the template tree, and quickly execute optimization strategies. This incremental log analysis method can achieve multiple functions, including log classification and construction of new templates, making it more flexible and efficient for emerging log patterns. The word vector model is regularly retrained and updated to maintain its adaptability to new data.

[0131] The disclosed embodiment combines a word vector model, a word frequency tree structure optimization, an intelligent pruning strategy, an efficient query acceleration mechanism, and a dynamic online processing capability to achieve efficient parsing, classification, and analysis of large-scale log data. The word vector model is used to better capture the semantic relationship of log data, and the contextual relationship of words is learned by training the word vector model to achieve the semantic similarity calculation of log template keywords. A multi-level pruning strategy is adopted, including pruning based on the threshold of the number of child nodes and the threshold of the similarity between sibling nodes, which can effectively filter out irrelevant log templates and improve the compactness of the template tree. The FT-tree after word vector optimization pruning has more accuracy and flexibility than the basic FT-tree in comprehensive performance. Configuring a fail pointer for each node in the template tree significantly speeds up the query speed, especially when processing long string matching, and improves the overall processing efficiency. By identifying and merging paths containing only a single child node, the tree structure is streamlined, memory usage is reduced, and the response speed and resource utilization of the system are further improved. A processing mechanism that can adapt to real-time log streams is designed, which can continuously integrate new log data and dynamically update word vector models and frequent template trees, ensuring the system's immediate adaptability to changes in log patterns and continuous and efficient analysis capabilities.

[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0133] Based on the same inventive concept, the embodiment of the present application also provides a log processing device for implementing the log processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more log processing device embodiments provided below can refer to the limitations on the log processing method above, and will not be repeated here.

[0134] In an exemplary embodiment, Figure 6 As shown, a log processing device 600 is provided, including:

[0135] The acquisition module 610 is used to acquire the log to be processed, and extract the target word segmentation in the log to be processed to obtain the word segmentation group to be processed;

[0136] The conversion module 620 is used to convert the word group to be processed by using a preset log template tree to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arrangement of the preset log based on a preset word database, and the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segment in the log.

[0137] In one embodiment, the device further includes a module for acquiring the preset word database, and the module for acquiring the preset word database is further used for:

[0138] Obtain an initial log, and divide the initial log to obtain a plurality of initial participles, each of the initial participles corresponding to a word frequency;

[0139] Determine the initial segmented words whose word frequency is greater than the preset word frequency as the preset segmented words;

[0140] Determining the association relationship between the preset participles according to the positions of the preset participles in the log;

[0141] According to the association relationship, the word vector corresponding to each of the preset word segments is determined to generate a preset word database.

[0142] In one of the embodiments, the device further includes a module for determining the preset log template tree, and the module for determining the preset log template tree is further used to:

[0143] Obtaining a preset log, and dividing the preset log to obtain multiple candidate segmentations;

[0144] Determine the candidate segmented words that match the preset word database as matching segmented words, and construct a word chain list according to the position and occurrence frequency of the matching segmented words in the preset log;

[0145] An initial log tree is generated according to the word chain list, and the nodes of the initial log tree are adjusted to obtain a preset log template tree.

[0146] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0147] Determine the number of child nodes corresponding to each node in the initial log tree;

[0148] Determine a node whose number of child nodes is greater than or equal to a first preset node threshold as a deletion node, and delete the child nodes corresponding to the deletion node to obtain a candidate log tree;

[0149] Determine a node in the candidate log tree whose number of child nodes is greater than a second preset node threshold and less than a first preset node threshold as a node to be pruned;

[0150] After processing the child nodes of the node to be pruned using the preset pruning strategy, a preset log template tree is obtained.

[0151] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0152] For each node to be pruned, determining a child node corresponding to the node to be pruned as a brother node;

[0153] Determine the similarity between the brother nodes according to the word vectors corresponding to the brother nodes;

[0154] When the similarity is greater than a preset threshold, a node whose similarity to the node to be pruned is less than a preset value is determined from the sibling nodes as a pruned node, and the pruned node is deleted to obtain a processed node to be pruned.

[0155] In one of the embodiments, the module for determining the preset log template tree is further used for:

[0156] For the adjusted initial log tree, set the pointer of the root node to be empty and the pointers of the direct child nodes of the root node to point to the root node;

[0157] For each node, search from the pointer of the parent node of the node to the leaf nodes step by step until the target node is determined, and set the pointer of the node to point to the target node, wherein the target node has the same child node as the node.

[0158] Each module in the above-mentioned log processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0159] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method described in this embodiment, such as logs to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log processing method is implemented.

[0160] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0162] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0163] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0165] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A log processing method, characterized in that: The method comprises: Obtaining a log to be processed, and extracting target segmented words in the log to be processed to obtain a segmented word group to be processed; The word segmentation group to be processed is converted using a preset log template tree to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arrangement of the preset log based on a preset word database, wherein the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segmentation in the log.

2. The method according to claim 1, characterized in that The method for acquiring the preset word database includes: Obtain an initial log, and divide the initial log to obtain a plurality of initial participles, each of the initial participles corresponding to a word frequency; Determine the initial segmented words whose word frequency is greater than the preset word frequency as the preset segmented words; Determining the association relationship between the preset participles according to the positions of the preset participles in the log; According to the association relationship, the word vector corresponding to each of the preset word segments is determined to generate a preset word database.

3. The method according to claim 1, characterized in that The method for determining the preset log template tree includes: Obtaining a preset log, and dividing the preset log to obtain multiple candidate segmentations; Determine the candidate segmented words that match the preset word database as matching segmented words, and construct a word chain list according to the position and occurrence frequency of the matching segmented words in the preset log; An initial log tree is generated according to the word chain list, and the nodes of the initial log tree are adjusted to obtain a preset log template tree.

4. The method according to claim 3, characterized in that After adjusting the nodes of the initial log tree, a preset log template tree is obtained, including: Determine the number of child nodes corresponding to each node in the initial log tree; Determine a node whose number of child nodes is greater than or equal to a first preset node threshold as a deletion node, and delete the child nodes corresponding to the deletion node to obtain a candidate log tree; Determine a node in the candidate log tree whose number of child nodes is greater than a second preset node threshold and less than a first preset node threshold as a node to be pruned; After processing the child nodes of the node to be pruned using the preset pruning strategy, a preset log template tree is obtained.

5. The method according to claim 4, characterized in that The using a preset pruning strategy to process the child nodes of the node to be pruned includes: For each node to be pruned, determining a child node corresponding to the node to be pruned as a brother node; Determine the similarity between the brother nodes according to the word vectors corresponding to the brother nodes; When the similarity is greater than a preset threshold, a node whose similarity to the node to be pruned is less than a preset value is determined from the sibling nodes as a pruned node, and the pruned node is deleted to obtain a processed node to be pruned.

6. The method according to claim 3, characterized in that: After adjusting the nodes of the initial log tree, the method further includes: For the adjusted initial log tree, set the pointer of the root node to be empty and the pointers of the direct child nodes of the root node to point to the root node; For each node, search from the pointer of the parent node of the node to the leaf nodes step by step until the target node is determined, and set the pointer of the node to point to the target node, wherein the target node has the same child node as the node.

7. A log processing device, characterized in that: The device comprises: An acquisition module is used to acquire the log to be processed, and extract the target word segmentation in the log to be processed to obtain the word segmentation group to be processed; A conversion module is used to convert the word group to be processed using a preset log template tree to obtain a target log that conforms to the preset log template tree, wherein the preset log template tree is obtained after word segmentation screening and arranging the preset log based on a preset word database, and the preset word database stores preset word segments and word vectors corresponding to the preset word segments, and the word vectors are determined based on the position of the preset word segment in the log.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images