Log clustering method, device, equipment and storage medium

Abstract

This application relates to the field of computer information processing and discloses a log clustering method, apparatus, device, and storage medium, comprising: acquiring target log text; extracting local features of the target log text and mapping them to a feature space to determine the corresponding target mapping position; recalling historical logs that are also mapped to the target mapping position from historical log data as a candidate log set; determining an adaptive distance threshold for similarity judgment based on the length feature of the target log text; calculating the matching distance between the target log text and each candidate log in the candidate log set, and if the matching distance is less than or equal to the adaptive distance threshold, determining that the target log text and the corresponding candidate log belong to the same cluster category. This method can be applied to log processing and system monitoring scenarios in business areas such as finance, insurance, and healthcare, and can improve the computational efficiency and clustering accuracy for processing massive heterogeneous logs.

Description

Technical Field

[0001] This application relates to the field of computer information processing and can be applied to business areas such as finance, insurance, and healthcare. In particular, it relates to a log clustering method, apparatus, device, and storage medium. Background Technology

[0002] In business systems such as financial insurance and healthcare, daily operations generate massive amounts of system alarms and business logs. To achieve root cause analysis, system monitoring, and anomaly detection, log clustering techniques are typically used to cluster the raw logs into representative log categories. Currently, commonly used log clustering techniques are mostly based on full text comparison algorithms for similarity matching. That is, during calculation, the log to be processed is compared text-by-text with all historical logs or cluster centers, and a fixed edit distance threshold or text similarity threshold is used to determine whether the logs belong to the same category.

[0003] The inventors recognized that existing log clustering schemes have the following specific drawbacks: On the one hand, when processing massive amounts of logs, the method of comparing each log one by one will generate a huge number of comparisons, resulting in severe consumption of computing resources and long clustering time, with obvious computing efficiency bottlenecks; on the other hand, the logs generated by different business modules have large differences in length and structure. Using a fixed similarity threshold for judgment cannot adapt to log features of different lengths. It is very easy to mistakenly merge logs with similar lengths but different key business parameters, or fail to accurately identify similar logs due to differences in field order. Ultimately, the existing schemes suffer from low computing efficiency and poor clustering accuracy. Summary of the Invention

[0004] This invention provides a log clustering method, apparatus, device, and storage medium to solve the technical problem that existing log clustering schemes suffer from low computational efficiency and poor clustering accuracy when processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds.

[0005] To achieve the above objectives, a first aspect of this application proposes a log clustering method, the method comprising: Obtain the target log text; Extract local features from the target log text and map them to the feature space to determine the corresponding target mapping position; From historical log data, historical logs that are also mapped to the target mapping location are used as a candidate log set for recall; Based on the length feature of the target log text, an adaptive distance threshold for similarity determination is determined. Calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, then determine that the target log text and the corresponding candidate log belong to the same cluster category.

[0006] To achieve the above objectives, a second aspect of this application provides a log clustering apparatus, the apparatus comprising: The acquisition module is used to acquire the target log text; The extraction module is used to extract local features of the target log text and map them to the feature space to determine the corresponding target mapping position; The recall module is used to recall historical logs that are also mapped to the target mapping location from historical log data as a candidate log set; The determination module is used to determine an adaptive distance threshold for similarity determination based on the length feature of the target log text; The clustering module is used to calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, the target log text and the corresponding candidate log belong to the same cluster category.

[0007] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0009] The above-mentioned log clustering method, apparatus, device, and storage medium can obtain the target log text; extract local features of the target log text and map them to the feature space to determine the corresponding target mapping position; recall historical logs that are also mapped to the target mapping position from historical log data as a candidate log set; determine an adaptive distance threshold for similarity judgment based on the length feature of the target log text; calculate the matching distance between the target log text and each candidate log in the candidate log set respectively; if the matching distance is less than or equal to the adaptive distance threshold, it is determined that the target log text and the corresponding candidate log belong to the same cluster category. In this application, for massive log processing and system monitoring scenarios in business areas such as finance, insurance, and healthcare, a two-stage clustering mechanism based on spatial mapping and dynamic thresholds can be used. First, the local features of the target log are extracted and spatially mapped to locate the target position, and historical logs at the same mapped position are recalled to form a candidate set. Spatial feature collision is used to replace the traversal comparison of the entire database, which greatly reduces the data scale of subsequent calculations. Then, the distance threshold is adaptively adjusted according to the length feature of the log text to obtain a judgment standard reflecting the current log structure. This dynamic threshold is then introduced as a condition in the distance comparison process with candidate logs, so that the clustering logic can effectively adapt to heterogeneous logs with different length features, improve the flexibility of log category classification and the overall matching degree, and solve the technical problems of low computational efficiency and poor clustering accuracy when processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds in existing log clustering schemes. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a log clustering method according to an embodiment of the present invention; Figure 2 This is a flowchart of the log clustering method provided in the embodiments of this application; Figure 3 This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 4 This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 5 This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 6This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 7 This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 8 This is another flowchart of the log clustering method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the log clustering device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] In business systems such as financial insurance and healthcare, daily operations generate massive amounts of system alarms and business logs. To achieve root cause analysis, system monitoring, and anomaly detection, log clustering techniques are typically used to cluster the raw logs into representative log categories. Currently, commonly used log clustering techniques are mostly based on full text comparison algorithms for similarity matching. That is, during calculation, the log to be processed is compared text-by-text with all historical logs or cluster centers, and a fixed edit distance threshold or text similarity threshold is used to determine whether the logs belong to the same category.

[0016] The inventors recognized that existing log clustering schemes have the following specific drawbacks: On the one hand, when processing massive amounts of logs, the method of comparing each log one by one will result in a huge number of comparisons, leading to severe consumption of computing resources and long clustering time, resulting in a significant bottleneck in computing efficiency; on the other hand, the logs generated by different business modules have large differences in length and structure. Using a fixed similarity threshold for judgment cannot adapt to log features of different lengths. It is very easy to mistakenly merge logs with similar lengths but different key business parameters, or fail to accurately identify similar logs due to differences in field order. Ultimately, this leads to low computing efficiency and poor clustering accuracy in existing schemes.

[0017] Based on this, embodiments of this application provide a log clustering method, apparatus, device, and storage medium, aiming to solve the technical problem that existing log clustering schemes suffer from low computational efficiency and poor clustering accuracy when processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds.

[0018] The log clustering method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the log clustering method in this application is described.

[0019] The log clustering method provided in this application embodiment can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can obtain the target log text to be processed through the client or log collection nodes, extract local features of the target log text and map them to the feature space to determine the corresponding target mapping position; from historical log data, historical logs that are also mapped to the target mapping position are used as a candidate log set for recall; based on the length feature of the target log text, an adaptive distance threshold for similarity determination is determined; the matching distance between the target log text and each candidate log in the candidate log set is calculated respectively. If the matching distance is less than or equal to the adaptive distance threshold, the target log text and the corresponding candidate log belong to the same cluster category, and the generated clustering result or interactive prompts based on the clustering result are fed back to the client. This application addresses massive log processing and system monitoring scenarios in business areas such as finance, insurance, and healthcare. It utilizes a two-stage clustering mechanism based on spatial mapping and dynamic thresholds. First, local features of the target log are extracted and spatially mapped to locate the target position. Historical logs at the same mapped position are directly recalled to form a candidate set. Spatial feature collision replaces the traversal and comparison of the entire database, significantly reducing the data scale of subsequent calculations. Then, the distance threshold is adaptively adjusted based on the length characteristics of the log text to derive a judgment standard reflecting the current log structure. This dynamic threshold is then introduced as a condition in the distance comparison process with candidate logs, enabling the clustering logic to effectively adapt to heterogeneous logs with different length characteristics. This improves the flexibility of log category classification and overall matching accuracy, solving the technical problems of low computational efficiency and poor clustering accuracy in processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds in existing log clustering schemes. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Specific embodiments of this application are described in detail below.

[0020] Figure 2 This is an optional flowchart of the log clustering method provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, steps S201 to S205.

[0021] Step S201: Obtain the target log text.

[0022] Step S202: Extract local features of the target log text and map them to the feature space to determine the corresponding target mapping position.

[0023] Step S203: From the historical log data, retrieve the historical logs that are also mapped to the target mapping location as a candidate log set.

[0024] Step S204: Based on the length feature of the target log text, determine the adaptive distance threshold for similarity determination.

[0025] Step S205: Calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, then determine that the target log text and the corresponding candidate log belong to the same cluster category.

[0026] In step S201 of some embodiments, the target log text refers to the single log data to be processed at the current moment that needs to undergo cluster analysis. In specific practical application scenarios, such as the daily operation of agent recruitment systems in the financial insurance field or hospital information systems (HIS) in the healthcare field, the corresponding network and hardware devices continuously generate a large number of log streams. In this step, log data generated can be acquired in real time or periodically in batches from the raw log stream through relevant data acquisition interfaces, and the log data to be processed at the moment is used as the target log text to provide basic data input conditions for subsequent cluster analysis.

[0027] Please see Figure 3 In some embodiments, steps S301 to S303 may be included before step S201.

[0028] Step S301: Obtain the original log data and parse the original log data according to the preset word segmentation rules to obtain a word sequence composed of multiple basic words.

[0029] Step S302: Based on the preset entity recognition model, scan the word sequence to identify dynamic variable segments representing time, network address, and variable values.

[0030] Step S303: Replace the dynamic variable fragment with the corresponding mask symbol, and reassemble the word sequence after replacing the mask symbol to generate the target log text.

[0031] In step S301 of some embodiments, raw log data refers to source log records that have not undergone any cleaning or structured preprocessing. In actual business environments, such as medical image archiving systems or core financial insurance business platforms, raw log data often contains a large number of unstructured long strings that mix business logic and underlying environmental information. To facilitate subsequent accurate feature extraction, these long strings first need to be finely segmented. According to preset word segmentation rules, such as regular expression matching rules based on specific delimiters such as spaces, hyphens, colons, etc., or word segmentation algorithms based on industry-specific dictionaries, the raw log data is segmented into the smallest indivisible semantic units, i.e., basic words. After parsing, the originally continuous character text is converted into a discrete word sequence with a logical order, establishing the basic data structure for subsequent variable identification and text cleaning.

[0032] In step S302 of some embodiments, in high-concurrency business scenarios such as healthcare or financial insurance, the timestamps, request node IP addresses, randomly generated transaction serial numbers, or device parameters attached to logs of the same business type are usually dynamically changing each time they are generated. Retaining these dynamic variables would cause logs with the same business essence to exhibit significant differences in textual form, thereby increasing noise in the feature mapping process and affecting the accuracy of the final clustering calculation. Therefore, this embodiment introduces a preset entity recognition model, such as a regular expression matching model based on a rule engine or a named entity recognition model, to scan the lexical sequence generated in step S301 one by one. This model can identify dynamic variable fragments in the lexical sequence that represent specific occurrence times, source or destination network addresses, and random variable values, which are not helpful in determining the business essence of the logs, thereby accurately locating the interfering features that need to be cleaned.

[0033] In step S303 of some embodiments, after identifying the aforementioned dynamic variable fragments, in order to maintain the original syntactic structure integrity of the log without disrupting the semantic relationship of the context, they are not directly deleted from the sequence. Instead, uniform mask symbols (e.g., introducing preset placeholder constants such as "", "", and "") are used to replace these dynamic variable fragments in situ. After the replacement is completed, the masked word sequence is reassembled according to its original order to form text with a standardized structure. This assembled clean text is then output as the "target log text" obtained in the aforementioned step S201 to the downstream feature mapping stage.

[0034] Through steps S301 to S303 described above, this embodiment introduces a pre-processing text cleaning and standardization step before formally performing local feature mapping. First, the original logs are parsed into word sequences using preset word segmentation rules. Then, an entity recognition model is used to locate dynamic variable fragments representing time, network addresses, and other factors that interfere with clustering decisions. Finally, in-situ replacement and reassembly are performed using a unified mask symbol to generate standardized target log text. This mechanism eliminates text noise interference caused by random environmental parameters and dynamic values, extracting a clean business structure template. This provides a high-quality input source for subsequent extraction of local features and calculation of mapping positions, preventing logs with the same business essence from being incorrectly classified into different categories due to fluctuations in random variables. This further improves the accuracy of the overall clustering scheme in scenarios with massive heterogeneous data.

[0035] In step S202 of some embodiments, after obtaining the target log text, local features of the target log text are extracted and mapped to a feature space to determine the corresponding target mapping position. Here, local features refer to feature data segments extracted from the target log text that can characterize its text structure or business content; the feature space refers to a mathematical space used to characterize the distribution of unstructured text features; and the target mapping position refers to the specific index position in the feature space where the local feature falls after processing by the corresponding feature mapping algorithm. Specifically, for massive and unstructured log data, directly comparing all text data would be computationally intensive. Therefore, this embodiment first extracts the local features of the target log text and uses a mapping mechanism to transform them into a multi-dimensional feature space. For example, in a healthcare scenario, when the obtained target log text is "Medical insurance settlement failed - insured person information inconsistent," the local features of this text can be extracted and mapped to specific positions such as hash buckets in the feature space. Through the above processing, unstructured text data can be converted into a data form that can be identified by spatial location, thereby determining the target mapping position corresponding to the target log text and providing a location basis for targeted retrieval in subsequent steps.

[0036] Please see Figure 4 In some embodiments, step S202 may include, but is not limited to, steps S401 to S403.

[0037] Step S401: Perform sequence parsing on the target log text, extract multiple local feature words that constitute the target log text, and construct a feature set of the target log text based on the multiple local feature words.

[0038] Step S402: Using a local sensitive hash cluster containing multiple independent hash functions, the feature set is subjected to dimensionality reduction mapping calculation to generate a signature vector of the target log text.

[0039] Step S403: Divide the signature vector into multiple sub-signature segments, and perform hash operations on each sub-signature segment to generate corresponding bucket index values. Use the hash bucket position pointed to by each bucket index value as the target mapping position.

[0040] In step S401 of some embodiments, local feature words refer to low-level text units with independent semantic or structural representation capabilities obtained after segmenting the target log text through a specific sliding window or word segmentation algorithm; the feature set is a mathematical set representation formed by aggregating and deduplicating the above words. After obtaining the target log text, in order to facilitate its transformation into a data form that can be mapped mathematically, it is necessary to first perform fine-grained feature decomposition. Specifically, the target log text is serialized from left to right according to preset parsing rules, extracting multiple consecutive local feature words, and combining these words to construct a feature set that fully represents the overall local features of the target log text. This step transforms the linear string text into a structured mathematical set form, laying the foundation for subsequent dimensionality reduction compression and hash mapping operations.

[0041] In step S402 of some embodiments, a Locality Sensitive Hash (LSH) cluster refers to a set of mapping algorithms composed of multiple completely independent hash functions, such as MurmurHash functions using different random seeds; the signature vector refers to a low-dimensional, fixed-length numerical sequence formed after the feature set has undergone the aforementioned hash mapping process. Since directly comparing data based on a high-dimensional feature set would result in huge computational resource consumption, this embodiment introduces a Locality Sensitive Hash (LSH) mechanism to reduce the dimensionality of high-dimensional features using multiple hash clusters. Let a Locality Sensitive Hash cluster containing K independent hash functions be denoted as… Let S be the feature set of the target log text. Then, substitute set S into each independent hash function in the hash cluster to calculate, thereby obtaining a set of independent hash values. These hash values ​​are then combined in sequence to generate the signature vector of the target log text. This dimensionality reduction mapping calculation can preserve the physical similarity relationship between the original local feature words to the greatest extent, so that similar feature sets have a very high probability of generating similar signature vectors.

[0042] In step S403 of some embodiments, a sub-signature segment refers to a shorter vector fragment obtained by dividing a long-dimensional signature vector according to a preset fixed step size; the bucket index value refers to the specific coordinates or identifier used for addressing obtained by performing spatial mapping calculation on each sub-signature segment again. To further amplify the collision probability of similar logs in the feature space, this embodiment divides the high-dimensional signature vector generated in step S402 into multiple non-overlapping sub-signature segments. Subsequently, a second hash modulo operation is independently performed on each sub-signature segment to calculate its corresponding bucket index value. Each bucket index value uniquely points to a specific hash bucket in the feature space, and the set of storage locations of these hash buckets together constitutes the "target mapping location" mentioned in step S202. Under this mechanism, as long as the signature vectors of two logs are completely identical in any sub-signature segment, they will be mapped to calculate the same bucket index value and fall into the same hash bucket location.

[0043] Through steps S401 to S403 described above, this embodiment first parses the target log text and constructs a feature set. Then, it uses a Local Sensitive Hash Cluster (LSH) containing multiple independent hash functions to perform dimensionality reduction calculations on this feature set to generate a fixed-length signature vector. Finally, the signature vector is divided into multiple sub-signature segments and mapped a second time to specific hash buckets, thereby obtaining the accurate target mapping position. This mechanism not only achieves efficient dimensionality reduction from high-dimensional text data to a low-dimensional feature space, but also effectively amplifies the probability of physical collisions between similar logs in the feature space using multiple hash mappings and segmentation mechanisms. This ensures that logs with high similarity can be mapped to the same hash bucket position, providing underlying feature localization and spatial partitioning for subsequent candidate set retrieval with high recall and low computational cost.

[0044] In step S203 of some embodiments, historical logs that are also mapped to the target mapping position are recalled from the historical log data as a candidate log set. During this process, the target mapping position determined in step S202 is used as the retrieval condition to search the historical log database where feature mapping has been completed. Since historical logs mapped to the same position have spatial collisions with the target log text in terms of local features, they have a high probability of text similarity at the physical feature level. Therefore, this embodiment extracts all historical logs that are also mapped to the target mapping position and combines them to form a candidate log set. Compared to the traversal method of full text comparison in the prior art, this step only uses historical logs that have a feature spatial mapping intersection with the target log text as candidate targets. For example, when processing hundreds of thousands of logs, this recall mechanism based on mapping position can significantly compress the amount of candidate log data. This mechanism effectively eliminates a large amount of obviously irrelevant log data, reduces the data scale of subsequent comparison calculations, and achieves pre-recall filtering of the data to be compared.

[0045] Please see Figure 5 In some embodiments, step S203 may include, but is not limited to, steps S501 to S503.

[0046] Step S501: For each hash bucket pointed to by the bucket index value in the target mapping location, extract the historical log identifier bound in the hash bucket.

[0047] Step S502: Count the collision frequency of each historical log identifier in all hash buckets where mapping hits occur, and remove historical log identifiers with a collision frequency lower than a preset collision threshold.

[0048] Step S503: Based on the retained historical log identifiers, extract the corresponding full log text from the historical log data, and perform merging and deduplication processing to generate a candidate log set.

[0049] In step S501 of some embodiments, the hash bucket to which the sub-signature segment mapping hits refers to the specific feature space storage unit pointed to by the bucket index value after hashing each sub-signature segment of the target log text in the previous mapping stage step S403; the historical log identifier refers to the identity code used to uniquely indicate the storage location or identity information of historical log data, such as a globally unique log ID. Specifically, after the sub-signature segments of the target log text are mapped to specific hash buckets (e.g., bucket 1, bucket 3, or bucket 5), these hash buckets have pre-stored historical log records with the same sub-signature segment characteristics. In this step, based on the located hash bucket position, all historical log identifiers stored in these hash buckets are read and extracted, thereby initially locking in potentially similar historical data that physically collide with the target log text in the feature space.

[0050] In step S502 of some embodiments, the collision frequency refers to the total number of times a specific historical log identifier appears repeatedly in the multiple hash buckets currently hit; the preset collision threshold is the minimum number of occurrences set to filter out accidental spatial overlaps. Although the locality-sensitive hashing mechanism can effectively recall similar data, due to the dimensionality reduction and compression characteristics of hash mapping, occasional hash collisions may cause a few business-irrelevant logs to falsely collide in individual hash buckets. If a historical log identifier is denoted as... The collision frequency obtained from the statistics is denoted as C. The preset collision threshold is denoted as Then, through mathematical comparison, it is determined that when C... If the historical log entries overlap only occasionally in a very small number of feature segments, then the corresponding historical log identifiers are removed from the list of potentially similar entries. This cross-validation mechanism based on multi-bucket frequency statistics effectively filters out noisy data introduced by hash dimensionality reduction collisions, improving the accuracy of candidate data.

[0051] In step S503 of some embodiments, after completing the noise filtering based on collision frequency, the remaining historical log identifiers represent reliable objects with a high probability of structural similarity to the target log text. Based on these retained historical log identifiers, a precise retrieval is performed in the historical log database or full storage medium to retrieve and extract the corresponding actual "full log text" (i.e., unstructured content containing complete business information and structure). Since the same historical log identifier may be referenced and recalled multiple times by multiple hash buckets during the statistical process, and there may also be redundant records with completely identical text structures in the database, after extracting the full log text, a merging and deduplication process needs to be performed to eliminate duplicates within the set. Finally, the deduplicated independent historical log texts are combined to generate a candidate log set for subsequent fine-grained distance matching.

[0052] Through steps S501 to S503 described above, this embodiment first extracts the bound historical log identifiers from the specific hash buckets hit by the mapping, and then eliminates noise interference items below a preset collision threshold by statistically analyzing the collision frequency of these identifiers in multiple buckets. Finally, it extracts the full log text based on the retained high-purity identifiers and performs merging and deduplication to construct the final candidate log set. This mechanism, through joint recall of multiple feature segments combined with frequency filtering, effectively eliminates false data with hash collisions associated with spatial mapping while ensuring a high recall rate for similar logs, further compressing the data size of the candidate set and providing high-quality, small-volume data for subsequent fine matching calculations based on adaptive distance thresholds.

[0053] In step S204 of some embodiments, in actual business scenarios such as finance, insurance, and healthcare, log data is diverse in type and varies significantly in length, including both short database operation logs and longer medical record parsing logs. Using a fixed similarity threshold often makes it difficult to accommodate heterogeneous logs of different lengths, easily leading to misjudgments such as irrelevant logs being incorrectly clustered or similar logs being missed. Therefore, this embodiment extracts the length feature of the target log text, such as the total number of characters, and dynamically calculates the corresponding similarity threshold accordingly. If the length feature of the target log text is denoted as L, and the adaptive distance threshold used for similarity determination is denoted as Threshold, it can be mathematically expressed using the following piecewise function rule: when L is less than a preset length threshold (e.g., 50), Threshold is set to a smaller preset fixed value (e.g., 3); when L is greater than or equal to this length threshold, it is dynamically calculated by combining a preset scaling factor (e.g., 8%) and a preset upper distance limit (e.g., 4). The mathematical description of the threshold generation can be expressed as:

[0054] By extracting length features and performing dynamic calculations, an adaptive distance threshold reflecting the structural characteristics of each target log text can be generated, enabling the similarity judgment criteria to dynamically adapt to changes in text length.

[0055] Please see Figure 6 In some embodiments, step S204 may include, but is not limited to, steps S601 to S604.

[0056] Step S601: Count the total number of tokens in the target log text, and use it as the length feature of the target log text.

[0057] Step S602: If the length feature is less than the preset length boundary value, then the adaptive distance threshold is set to the preset fixed threshold.

[0058] Step S603: If the length feature is greater than or equal to the preset length boundary value, calculate the product of the length feature and the preset tolerance ratio factor, and determine the dynamic threshold based on the product result.

[0059] Step S604: Compare the dynamic threshold with the preset distance upper limit and select the smaller value as the final adaptive distance threshold for similarity determination.

[0060] In step S601 of some embodiments, the number of basic terms contained in the target log text after the aforementioned parsing process is traversed and accumulated. This statistical result directly reflects the physical scale of the current log to be matched, and serves as the quantitative basis for the aforementioned length feature L, providing basic data input for subsequent segmentation judgment logic.

[0061] In step S602 of some embodiments, after obtaining the length feature L, it is numerically compared with a pre-configured length threshold. When it is determined that L is in the range less than the threshold, it indicates that the current object belongs to the short text form. Since small word differences within short texts usually correspond to substantial changes in business logic, this branch logic is executed to directly assign a small fixed threshold to the adaptive distance threshold Threshold, thereby strictly limiting the tolerance for text differences within this range.

[0062] In step S603 of some embodiments, when the comparison determination L is greater than or equal to a preset length threshold, the long text processing logic is triggered. At this time, a multiplication operation is performed to multiply the obtained length feature L by a preset tolerance ratio factor to obtain the corresponding dynamic threshold. This operation allows the allowable difference range of long text features during matching to expand linearly with its own length, so as to match reasonable structural fluctuations in longer text content.

[0063] In step S604 of some embodiments, to prevent excessively long text from causing the calculated dynamic threshold to be too large due to proportional multiplication, thereby leading to mismatches across business categories, truncation logic is introduced after step S603. Specifically, the calculated dynamic threshold is compared with a preset upper limit value for distance, and the minimum value between the two is extracted. Through this minimum value selection operation, the adaptive distance threshold is constrained within an objectively reasonable error convergence range.

[0064] Through the above steps S601 to S604, this embodiment of the application, based on the total number of tokens in the target log text, performs fixed threshold assignment or dynamic ratio calculation combined with upper limit truncation operation logic through conditional branch comparison, and outputs a judgment threshold that has both dynamic adaptability and boundary constraints without manual intervention, thus establishing a reliable comparison benchmark for subsequent accurate distance calculation.

[0065] In step S205 of some embodiments, after obtaining the candidate log set and determining the adaptive distance threshold, for each candidate log in the candidate log set, a text difference metric, such as edit distance, is calculated between it and the target log text to obtain the corresponding matching distance. Subsequently, each matching distance is compared with the adaptive distance threshold calculated in step S204. For example, in a medical business scenario, for the target log text "Patient Allergy History Not Updated - Penicillin" and the candidate log "Patient Allergy History Not Updated - Cephalosporin," the matching distance between them is calculated. If the matching distance is less than or equal to the adaptive distance threshold corresponding to the target log text, it indicates that the difference in text morphology between the two logs is within a reasonable fluctuation range allowed by the current length feature, and they have consistent business attributes. Therefore, it is determined that the target log text and the corresponding candidate log belong to the same cluster category. This step, by limiting the calculation scope to the candidate log set and introducing adaptive distance judgment conditions, ensures the objectivity and accuracy of the final clustering result.

[0066] Please see Figure 7 In some embodiments, the step of calculating the matching distance between the target log text and each candidate log in the candidate log set in step S205 may include, but is not limited to, steps S701 to S703.

[0067] Step S701: Construct a dynamic programming matrix between the target log text and the currently compared candidate logs.

[0068] Step S702: Calculate the state values ​​in the dynamic programming matrix layer by layer according to the preset hierarchical traversal order, and after completing the calculation of the state values ​​of the current layer, extract the minimum state value of the current layer as the intermediate distance value.

[0069] Step S703: If the intermediate distance value is greater than the adaptive distance threshold, the calculation of the remaining state values ​​in the dynamic programming matrix is ​​stopped, and it is determined that the target log text and the currently compared candidate log do not belong to the same cluster category.

[0070] In step S701 of some embodiments, the dynamic programming matrix refers to a two-dimensional mathematical state space used to record the transformation costs of each subsequence when calculating the edit distance between two text sequences. Since the target log text and the candidate logs in the candidate log set may differ in text length and word arrangement, to accurately quantify this structural difference, the word sequences of both need to be mapped to the rows and columns of this two-dimensional matrix respectively. In actual execution, for the target log text and a candidate log currently being compared, a blank dynamic programming matrix of the corresponding size is initialized. Each coordinate node in the matrix corresponds to the cost distribution accumulated by the edit operations between a prefix substring of the target log text and a prefix substring of the candidate log. This step, by constructing the dynamic programming matrix, establishes the basic computational space for subsequent refined calculation of character-level or word-level physical differences between log texts.

[0071] In step S702 of some embodiments, the preset hierarchical traversal order typically refers to the calculation path that progresses step by step from the top left to the bottom right along the rows or diagonals of the matrix; the state value represents the accumulated edit distance value at the current node. Traditional edit distance calculation requires traversing and filling the entire matrix to obtain the final result, but in high-concurrency matching scenarios with massive logs, such a full traversal consumes a lot of computing power. Therefore, in this step, according to the hierarchical traversal order, after updating the state values ​​of all nodes in the current row or column, a minimum value operation is performed to extract the state value with the smallest value in that level, which is defined as the intermediate distance value. Mathematically, this intermediate distance value represents the minimum possible error between the target log text and the candidate log as of the current comparison progress, and is the lower bound of the subsequent state transition calculation.

[0072] In step S703 of some embodiments, due to the monotonically increasing nature of dynamic programming, the state value of subsequent computational levels must be greater than or equal to the minimum state value of the current level. Therefore, the intermediate distance value extracted in step S702 is numerically compared with the adaptive distance threshold determined in the aforementioned steps. Once it is found that the current intermediate distance value is greater than the adaptive distance threshold, it can be directly inferred from mathematical logic that regardless of how the remaining uncomputed nodes in the matrix undergo state transitions, the final calculated complete edit distance will inevitably be greater than the adaptive distance threshold. Based on this inference, a pruning mechanism is immediately triggered to stop redundant computation operations on the remaining grid nodes in the dynamic programming matrix, and a conclusion that the target log text and the candidate log do not belong to the same cluster is directly output.

[0073] Through steps S701 to S703 described above, this embodiment first provides a low-level comparison environment by constructing a two-dimensional state space. Then, during the process of progressively deducing state values, it continuously extracts intermediate distance values ​​representing the lower bound of the error, and terminates the calculation when this lower bound exceeds the adaptive distance threshold. This mechanism, while ensuring the mathematical rigor of text similarity calculation, effectively avoids invalid traversal calculations for long and dissimilar logs, significantly reducing the computational power consumption of the fine-matching stage and improving the processing throughput of a single log in the comparison phase.

[0074] Please see Figure 8 In some embodiments, step S205, which calculates the matching distance between the target log text and each candidate log in the candidate log set, may be followed by, but is not limited to, steps S801 to S803.

[0075] Step S801: If it is determined that the matching distance between the target log text and all candidate logs in the candidate log set is greater than the adaptive distance threshold, then an independent clustering identifier is assigned to the target log text.

[0076] Step S802: Based on the clustering identifier, the mapping relationship of each sub-signature segment of the target log text is synchronously constructed into the index of the corresponding hash bucket.

[0077] Step S803: The target log text is used as the baseline feature data for the new clustering category and is asynchronously and persistently stored in the historical log data.

[0078] In step S801 of some embodiments, when the target log text performs dynamic programming distance calculations with each candidate log in the candidate log set generated in step S503, and the determination results are all mismatches, it indicates that the target log text represents a completely new log format that has never appeared before in terms of physical structure and business logic. For this global miss phenomenon, a globally unique independent code, namely a cluster identifier, is generated. This identifier is used to logically establish the target log text as a completely new cluster category.

[0079] In step S802 of some embodiments, to ensure that this new cluster category can be successfully recalled from the continuously generated massive logs, the underlying feature mapping space needs to be updated. Specifically, the sub-signature segments and their corresponding bucket index values ​​calculated in step S403 of the target log text are extracted, and the newly allocated cluster identifier is bound to these specific hash bucket storage structures. Through this reverse indexing operation, the local feature fragments of the new category log are registered in the retrieval space of locality-sensitive hashing, so that newly added logs with similar features in the future can be matched with the cluster identifier through hash dimensionality reduction mapping, thereby maintaining the continuous effectiveness of the candidate set recall mechanism.

[0080] In step S803 of some embodiments, the baseline feature data refers to template text representing the standard structural form of the new cluster category; asynchronous persistent storage refers to saving the data to the physical medium or database using a non-blocking background write method. After completing the index update in memory, the complete text content of the target log text and its associated cluster identifier are packaged and defined as the comparison baseline for the new category. To avoid the I / O latency caused by the disk write operation blocking the main thread of the clustering processing of the current real-time log stream, an asynchronous write strategy is used to append the baseline feature data to the historical log data pool for use in subsequent steps of extracting new log data and generating candidate sets.

[0081] Through steps S801 to S803 described above, this embodiment of the application, after confirming that the target log text does not belong to any known historical category, assigns an independent clustering identifier and synchronously updates the sub-signature segment index within the hash bucket to enable subsequent similarity recall capabilities. Finally, the log text is asynchronously persisted to disk as baseline feature data. This mechanism enables the overall log clustering scheme to possess self-evolution and knowledge base expansion capabilities under unsupervised conditions. It can automatically adapt to new log formats that continuously emerge with business iterations without manual intervention, ensuring the topological integrity of the heterogeneous log monitoring platform during long-term operation.

[0082] As can be seen, in the above solution, for massive log processing and system monitoring scenarios in business areas such as finance, insurance, and healthcare, a two-stage clustering mechanism based on spatial mapping and dynamic thresholds can be used. First, the local features of the target log are extracted and spatially mapped to locate the target position, and historical logs at the same mapped position are recalled to form a candidate set. Spatial feature collision is used to replace the traversal comparison of the entire database, which greatly reduces the data scale of subsequent calculations. Then, the distance threshold is adaptively adjusted according to the length feature of the log text to obtain a judgment standard reflecting the current log structure. This dynamic threshold is then introduced as a condition in the distance comparison process with candidate logs, so that the clustering logic can effectively adapt to heterogeneous logs with different length features, improve the flexibility of log category classification and the overall matching degree, and solve the technical problems of low computational efficiency and poor clustering accuracy when processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds in existing log clustering schemes.

[0083] Please see Figure 9 This application also provides a log clustering apparatus that can implement the above-described log clustering method. The apparatus includes: The acquisition module is used to acquire the target log text; The extraction module is used to extract local features of the target log text and map them to the feature space to determine the corresponding target mapping location; The recall module is used to recall historical logs that are also mapped to the target mapping location from the historical log data as a candidate log set; The determination module is used to determine an adaptive distance threshold for similarity judgment based on the length feature of the target log text; The clustering module is used to calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, the target log text and the corresponding candidate log are determined to belong to the same cluster category.

[0084] In some embodiments, the extraction module is further configured to: Obtain the raw log data and parse it according to the preset word segmentation rules to obtain a word sequence composed of multiple basic words; Based on a pre-defined entity recognition model, the system scans word sequences to identify dynamic variable segments representing time, network address, and variable values. The dynamic variable fragments are replaced with the corresponding mask symbols, and the word sequences after the mask symbols are replaced are reassembled to generate the target log text.

[0085] In some embodiments, the extraction module is specifically used for: Sequence parsing is performed on the target log text to extract multiple local feature words that constitute the target log text, and a feature set of the target log text is constructed based on the multiple local feature words. A local sensitive hash cluster containing multiple independent hash functions is used to perform dimensionality reduction mapping calculation on the feature set to generate a signature vector of the target log text; The signature vector is divided into multiple sub-signature segments, and each sub-signature segment is hashed to generate a corresponding bucket index value. The hash bucket position pointed to by each bucket index value is used as the target mapping position.

[0086] In some embodiments, the recall module is specifically used for: For each hash bucket pointed to by the bucket index value in the target mapping location, extract the historical log identifier bound in the hash bucket; Count the collision frequency of each historical log identifier in all hash buckets where mapping matches occur, and remove historical log identifiers with a collision frequency lower than a preset collision threshold; Based on the retained historical log identifiers, the corresponding full log text is extracted from the historical log data, and then merged and deduplicated to generate a candidate log set.

[0087] In some embodiments, the determining module is specifically used for: The total number of tokens in the target log text is counted and used as the length feature of the target log text. If the length feature is less than the preset length threshold, the adaptive distance threshold will be set to the preset fixed threshold. If the length feature is greater than or equal to the preset length threshold, the product of the length feature and the preset tolerance ratio factor is calculated, and the dynamic threshold is determined based on the product result. The dynamic threshold is compared with the preset distance upper limit, and the smaller value is selected as the final adaptive distance threshold used for similarity determination.

[0088] In some embodiments, the clustering module is specifically used for: Construct a dynamic programming matrix between the target log text and the currently compared candidate logs; The state values ​​in the dynamic programming matrix are calculated layer by layer according to the preset hierarchical traversal order, and after the calculation of the state value of the current layer is completed, the minimum state value of the current layer is extracted as the intermediate distance value. If the intermediate distance value is greater than the adaptive distance threshold, the calculation of the remaining state values ​​in the dynamic programming matrix is ​​stopped, and it is determined that the target log text and the currently compared candidate log do not belong to the same cluster category.

[0089] In some embodiments, the clustering module is further configured to: If it is determined that the matching distance between the target log text and all candidate logs in the candidate log set is greater than the adaptive distance threshold, then an independent clustering identifier is assigned to the target log text. Based on clustering identifiers, the mapping relationship of each sub-signature segment of the target log text is synchronously constructed into the index of the corresponding hash bucket; The target log text is used as the baseline feature data for the new clustering category and is asynchronously and persistently stored in the historical log data.

[0090] This invention provides a log clustering device. First, it extracts local features of the target log and performs spatial mapping to locate the target position. Then, it recalls historical logs at the same mapped position to form a candidate set. Spatial feature collision replaces the traversal and comparison of the entire database, significantly reducing the data scale of subsequent calculations. Next, it adaptively adjusts the distance threshold based on the length characteristics of the log text to derive a judgment standard reflecting the current log structure. This dynamic threshold is then introduced as a condition in the distance comparison process with candidate logs, enabling the clustering logic to effectively adapt to heterogeneous logs with different length characteristics. This improves the flexibility of log category division and the overall matching degree, and solves the technical problem of low computational efficiency and poor clustering accuracy when processing massive heterogeneous logs due to the use of full text comparison and fixed similarity calculation thresholds in existing log clustering schemes.

[0091] Specific limitations regarding the log clustering device can be found in the limitations of the log clustering method above, and will not be repeated here. Each module in the aforementioned log clustering device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0092] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a log clustering method on the server side.

[0093] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a client-side function or step based on a log clustering method.

[0094] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the target log text; Extract local features from the target log text and map them to the feature space to determine the corresponding target mapping position; From historical log data, historical logs that are also mapped to the target mapping location are used as a candidate log set for recall; Based on the length feature of the target log text, an adaptive distance threshold for similarity determination is determined; Calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, then determine that the target log text and the corresponding candidate log belong to the same cluster category.

[0095] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the target log text; Extract local features from the target log text and map them to the feature space to determine the corresponding target mapping location; From historical log data, historical logs that are also mapped to the target mapping location are used as a candidate log set for recall; Based on the length feature of the target log text, an adaptive distance threshold for similarity determination is determined; Calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, then determine that the target log text and the corresponding candidate log belong to the same cluster category.

[0096] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0100] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A log clustering method, characterized in that, include: Obtain the target log text; Extract local features from the target log text and map them to the feature space to determine the corresponding target mapping position; From historical log data, historical logs that are also mapped to the target mapping location are used as a candidate log set for recall; Based on the length feature of the target log text, an adaptive distance threshold for similarity determination is determined. Calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, then determine that the target log text and the corresponding candidate log belong to the same cluster category.

2. The log clustering method according to claim 1, characterized in that, Before obtaining the target log text, the method further includes: The raw log data is obtained and parsed according to the preset word segmentation rules to obtain a word sequence composed of multiple basic words; Based on a preset entity recognition model, the word sequence is scanned to identify dynamic variable segments representing time, network address, and variable values. The dynamic variable fragments are replaced with the corresponding mask symbols, and the word sequence after replacing the mask symbols is reassembled to generate the target log text.

3. The log clustering method according to claim 1, characterized in that, The step of extracting local features of the target log text and mapping them to the feature space to determine the corresponding target mapping position includes: The target log text is subjected to sequence parsing to extract multiple local feature words that constitute the target log text, and a feature set of the target log text is constructed based on the multiple local feature words. A local sensitive hash cluster containing multiple independent hash functions is used to perform dimensionality reduction mapping calculation on the feature set to generate the signature vector of the target log text; The signature vector is divided into multiple sub-signature segments, and a hash operation is performed on each sub-signature segment to generate a corresponding bucket index value. The hash bucket position pointed to by each bucket index value is used as the target mapping position.

4. The log clustering method according to claim 3, characterized in that, The step of recalling historical logs that are also mapped to the target mapping location from historical log data as a candidate log set includes: For each hash bucket pointed to by the bucket index value in the target mapping location, extract the historical log identifier bound in the hash bucket; The collision frequency of each historical log identifier in all hash buckets where mapping matches occur is counted, and historical log identifiers with a collision frequency lower than a preset collision threshold are removed. Based on the retained historical log identifiers, the corresponding full log text is extracted from the historical log data and merged and deduplicated to generate the candidate log set.

5. The log clustering method according to claim 1, characterized in that, The step of determining the adaptive distance threshold for similarity determination based on the length feature of the target log text includes: The total number of tokens in the target log text is counted and used as the length feature of the target log text. If the length feature is less than the preset length boundary value, then the adaptive distance threshold is set to the preset fixed threshold. If the length feature is greater than or equal to the preset length threshold, then the product of the length feature and the preset tolerance ratio factor is calculated, and the dynamic threshold is determined based on the product result. The smaller value is selected as the final adaptive distance threshold for similarity determination by comparing the dynamic threshold with the preset distance upper limit.

6. The log clustering method according to claim 1, characterized in that, The step of calculating the matching distance between the target log text and each candidate log in the candidate log set includes: Construct a dynamic programming matrix between the target log text and the currently compared candidate logs; The state values ​​in the dynamic programming matrix are calculated layer by layer according to the preset hierarchical traversal order, and after the calculation of the state value of the current layer is completed, the minimum state value of the current layer is extracted as the intermediate distance value. If the intermediate distance value is greater than the adaptive distance threshold, the calculation of the remaining state values ​​in the dynamic programming matrix is ​​stopped, and it is determined that the target log text and the currently compared candidate log do not belong to the same cluster category.

7. The log clustering method according to claim 3, characterized in that, After calculating the matching distance between the target log text and each candidate log in the candidate log set, the method further includes: If it is determined that the matching distance between the target log text and all candidate logs in the candidate log set is greater than the adaptive distance threshold, then an independent clustering identifier is assigned to the target log text; Based on the clustering identifier, the mapping relationship of each sub-signature segment of the target log text is synchronously constructed into the index of the corresponding hash bucket; The target log text is used as the baseline feature data for the new clustering category and is asynchronously and persistently stored in the historical log data.

8. A log clustering device, characterized in that, The device includes: The acquisition module is used to acquire the target log text; The extraction module is used to extract local features of the target log text and map them to the feature space to determine the corresponding target mapping position; The recall module is used to recall historical logs that are also mapped to the target mapping location from historical log data as a candidate log set; The determination module is used to determine an adaptive distance threshold for similarity determination based on the length feature of the target log text; The clustering module is used to calculate the matching distance between the target log text and each candidate log in the candidate log set. If the matching distance is less than or equal to the adaptive distance threshold, the target log text and the corresponding candidate log belong to the same cluster category.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the log clustering method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the log clustering method according to any one of claims 1 to 7.

Images