A method, apparatus, device, medium, and product for processing log data.

By dynamically adjusting the mask configuration and using clustering algorithms to process log data, the problem of distinguishing users from different sources of log data was solved, enabling accurate tracing of fault causes while protecting privacy and improving the accuracy of root cause analysis.

CN123114918APending Publication Date: 2026-11-17CHINA MOBILE INTERNET CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611435558.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies for log data processing, static and fixed mask configurations make it difficult to accurately distinguish users when performing correlation analysis between different sources, affecting the tracing of root causes of failures, and failing to balance user privacy protection with data availability.

Method used

By dynamically adjusting the mask configuration, grouping users according to their size and number of active users, using a clustering algorithm to evaluate the distinguishability, adjusting the mask parameters in real time to improve the distinguishability, and combining time synchronization processing and time-series aggregation, a user operation sequence is formed.

Benefits of technology

It achieves accurate merging of user operation records while protecting privacy, improves the accuracy of fault cause tracing, and dynamically balances privacy protection and fault tracing accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114918A_ABST
    Figure CN123114918A_ABST
Patent Text Reader

Abstract

This invention discloses a log data processing method, apparatus, device, medium, and product. It collects log data from several log sources, including mask parameters, the number of currently active users, mask identity information, and user operation events. Based on the number of active users and mask identity information, the log data is grouped according to user scale. The distinguishability between different users in each user group is evaluated to obtain a distinguishability score. For user identity information in user groups with a distinguishability score below a threshold, the target mask parameters are used to update the mask identity information. Based on the updated mask identity information, user operation events of the same user in different log sources are associated to form a user operation sequence, resulting in a user behavior dataset. This invention dynamically adjusts the mask configuration according to user scale, effectively balancing user privacy protection needs with cross-source log association, and improving the accuracy of log analysis and fault cause tracing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a log data processing method, apparatus, device, medium, and product. Background Technology

[0002] In the fields of information security and system operation and maintenance, log analysis is a key means of tracing the root causes of failures. With the widespread application of distributed systems and microservice architectures, a complete user operation often spans multiple log sources, such as application server logs, gateway logs, and database operation logs. Tracing the root causes of failures requires correlation analysis of log data scattered from different sources to reconstruct the user's operation trajectory.

[0003] In addition, log data contains a large amount of user privacy information, such as mobile phone numbers, ID card numbers, and accounts. In order to meet the requirements of privacy protection regulations, existing systems usually use masking technology to de-identify sensitive fields, that is, to retain some digits and hide the rest when outputting logs, in order to balance data availability and privacy security.

[0004] However, while masking improves security, it overlooks the complexities of cross-origin log data analysis. Because existing masking configurations are typically static, excessively large masking ranges can obscure key user information, leading to confusion when analyzing user behavior and causing the merging of different user errors during cross-source log consolidation, thus hindering accurate root cause identification. Conversely, insufficient masking ranges can expose too much sensitive user information. Especially in scenarios involving large user groups, relying solely on fixed masking rules struggles to adapt to dynamically changing user scale and behavioral patterns. Summary of the Invention

[0005] The purpose of this invention is to provide a log data processing method, apparatus, device, medium, and product that can dynamically adjust the mask configuration according to the user scale, effectively balancing user privacy protection needs with cross-source log association, and improving the accuracy of log analysis to enhance the accuracy of fault cause tracing.

[0006] To achieve the above objectives, embodiments of the present invention provide a log data processing method, comprising: Log data is collected from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; Based on the number of active users and the masked identity information, the log data is grouped according to user scale to generate several user groups, and the degree of distinguishability between different users in each user group is evaluated to obtain a distinguishability score. For user identity information in user groups whose discrimination scores are lower than a preset first score threshold, a target mask parameter is used to process the masked identity information to update the masked identity information; wherein, the target mask parameter is used to improve the discrimination score; Based on the updated mask identity information, user operation events of the same user in different log sources are associated to form a user operation sequence, thus obtaining a user behavior dataset.

[0007] As an improvement to the above scheme, the log data is grouped according to user scale based on the number of active users and the masked identity information to generate several user groups, and the distinguishability between different users in each user group is evaluated to obtain a distinguishability score, including: The number of active users and the masked identity information are correlated to generate a user scale feature vector; Based on the user size feature vector, a preset clustering algorithm is used to divide the log data into user groups of different user sizes; Compare the masked identity information of different users within each user group, and count the user pairs that cannot be effectively distinguished by the masked identity information, and record them as similar user pairs; The discrimination score of a user group is determined based on the proportion of the number of similar user pairs to the total number of user pairs within the user group.

[0008] As an improvement to the above scheme, the mask parameters include information on mask reserved bits; The step of comparing the masked identity information of different users within each user group and statistically analyzing user pairs that cannot be effectively distinguished by the masked identity information is denoted as similar user pairs, including: Form a user pair from any two users within the same user group; Compare the values ​​of the mask reserved bits in the mask identity information of the user pair bit by bit, and count the number of bits with the same value, which are recorded as the overlapping bits; When the number of overlapping bits exceeds a preset overlap threshold, the user pair is determined to be a similar user pair that cannot be effectively distinguished.

[0009] As an improvement to the above scheme, the user identity information in user groups whose discrimination scores are lower than a preset first score threshold is processed using target mask parameters to update the masked identity information, including: The user group whose discrimination score is lower than a preset first score threshold is identified and recorded as the target user group. Generate target mask parameters based on the total number of users in the target user group and the original mask parameters; The target mask parameters are used to process the user identity information in the target user group in order to update the mask identity information of the users in the target user group.

[0010] As an improvement to the above scheme, the mask parameters include information on mask reserved bits; The step of generating target mask parameters based on the total number of users in the target user group and the current mask parameters includes: Obtain the total number of users in the target user group, and calculate the theoretical number of bits required to distinguish this number of users based on the total number of users; Based on the difference between the theoretical number of bits to be retained and the current number of bits to be retained, the increment of the number of bits to be retained is determined, and the increment is added to the current number of bits to be retained to obtain the adjusted target number of bits to be retained.

[0011] As an improvement to the above scheme, the step of associating user operation events of the same user in different log sources based on the updated mask identity information to form a user operation sequence, thereby obtaining a user behavior dataset, includes: Using the time window as the classification dimension, log data from different log sources within the same time window are aggregated; User operation events that are within the same time window and have the same mask identity information are grouped into user operation events of the same user. The user operation events of the same user are sorted according to the corresponding operation timestamps to form the user operation sequence of the user; The user operation sequence is associated with and stored with the corresponding mask identity information and the source identifier of each user operation event to obtain a user behavior dataset.

[0012] As an improvement to the above solution, after collecting log data from several log sources, the method further includes: All the log data are synchronized in time to generate a log dataset in a uniform format; The time synchronization process includes: Using a preset standard time base as a reference, calculate the offset between the operation timestamp corresponding to the user operation event of each log data and the standard time base; The timestamp of each operation is compensated and corrected according to the offset to obtain the corrected log record.

[0013] As an improvement to the above scheme, the log data also includes fault events; After associating user operation events of the same user in different log sources based on the updated mask identity information to obtain a user behavior dataset, the method further includes: The cause of the failure is traced based on the user behavior dataset and the failure event.

[0014] As an improvement to the above solution, the step of tracing the cause of the fault based on the user behavior dataset and the fault event includes: Determine the fault time window based on the fault occurrence time in the fault event; Based on the user operation sequences in the user behavior dataset, filter the target operation sequences whose operation timestamps fall within the fault time window; The operation timestamps and source identifiers of each user operation event in the target operation sequence are compared with the fault trigger time and fault source in the fault event, respectively. The matching score is calculated based on the proportion of the number of events that match to the total number of events in the target operation sequence. Based on the target operation sequence whose matching score exceeds a preset second score threshold, generate the fault triggering reason.

[0015] As an improvement to the above solution, before tracing the cause of the fault based on the user behavior dataset and the fault event, the method further includes: Based on the user behavior dataset, the user's behavioral features are extracted, and a preset classification algorithm is used to distinguish the user into real users and fake users, resulting in user operation sequences with identity labels. The step of filtering target operation sequences whose operation timestamps fall within the fault time window based on the user operation sequences in the user behavior dataset specifically involves: Filter user action sequences labeled as real users from the user behavior dataset; From the user operation sequences of the real users, filter the target operation sequences whose operation timestamps fall within the fault time window.

[0016] As an improvement to the above scheme, the behavioral characteristics include at least one of operation frequency, operation time interval, number of cross-source switching, and operation type distribution.

[0017] As an improvement to the above scheme, the preset classification algorithm is a random forest model based on decision tree ensemble.

[0018] As an improvement to the above solution, the method further includes: Target operation sequences whose matching scores exceed a preset second score threshold are marked as high matching levels; The ratio of the number of user operation sequences with high matching levels to the number of target operation sequences is calculated as the tracing success rate. The mask parameters are adjusted based on the comparison between the traceability success rate and the preset success rate threshold.

[0019] This invention also provides a log data processing apparatus, comprising: The log data acquisition module is used to collect log data from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; The discrimination score calculation module is used to group the log data according to the user scale based on the number of active users and the mask identity information, generate several user groups, and evaluate the degree of distinguishability between different users in each user group to obtain a discrimination score. The mask parameter adjustment module is used to process the user identity information in the user group whose discrimination score is lower than a preset first score threshold using a target mask parameter to update the mask identity information; wherein, the target mask parameter is used to improve the discrimination score; The user behavior association module is used to associate user operation events of the same user in different log sources to form a user operation sequence based on the updated mask identity information, thereby obtaining a user behavior dataset.

[0020] This invention also provides a log data processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the log data processing method as described in any of the preceding embodiments.

[0021] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the log data processing method as described in any of the preceding embodiments.

[0022] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the log data processing method as described in any of the above embodiments.

[0023] Compared with existing technologies, the log data processing method, apparatus, device, medium, and product disclosed in this invention integrates multi-source log data, uses clustering algorithms to group users by size and evaluates the distinguishability of users within each group, and senses the effectiveness of the mask configuration under the current user size in real time, dynamically adjusting the mask parameters accordingly. For low-distinguishability groups, the mask parameters are dynamically adjusted to enhance the distinguishability, avoiding both excessive masking that makes user identities indistinguishable and insufficient masking that leads to privacy leaks. This achieves a dynamic balance between privacy protection strength and fault tracing accuracy. By uniformly processing multi-source logs with the adjusted mask parameters and combining them with time-series aggregation, it can accurately merge the operation records of the same user scattered across different log sources while protecting privacy, thus reconstructing the user's operation trajectory. This overcomes the problem of incompatibility between internal identifiers of different log sources, which leads to a lack of correlation, and is beneficial for tracing the root cause of faults, improving the accuracy of fault cause tracing. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a log data processing method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a preferred log data processing method in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0027] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0028] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0029] See Figure 1 This is a flowchart illustrating a log data processing method provided in an embodiment of the present invention. The embodiment of the present invention provides a log data processing method, including steps S11 to S14: S11. Collect log data from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; S12. Based on the number of active users and the mask identity information, the log data is grouped according to user scale to generate several user groups, and the distinguishability between different users in each user group is evaluated to obtain a distinguishability score. S13. For user identity information in user groups whose discrimination scores are lower than a preset first score threshold, a target mask parameter is used to process the mask identity information to update the mask identity information; wherein, the target mask parameter is used to improve the discrimination score. S14. Based on the updated mask identity information, associate the user operation events of the same user in different log sources to form a user operation sequence, and obtain a user behavior dataset.

[0030] In this embodiment of the invention, the log sources include application server logs, gateway logs, database operation logs, etc. A complete user operation often spans multiple log sources. This invention reconstructs the user's operation trajectory by performing correlation analysis on log data scattered from different sources. However, in the log data output by each log source, the user identity information is usually masked after being processed by the originally defined masking parameters. Different user identity information may result in the same masked identity information after masking, which may affect the accuracy of user behavior analysis.

[0031] Taking a specific business scenario as an example, in a multi-system operation and maintenance platform, after a failure occurs, it is necessary to analyze log data from multiple sources to trace the user operations that caused the problem. Assuming that user identity information in the logs is largely hidden, retaining only a few characteristic fields, then during peak periods when a large number of users are online simultaneously, the system may not be able to accurately distinguish which operations belong to the same user. This could lead to irrelevant operations being grouped together during fault analysis, ultimately resulting in incorrect conclusions.

[0032] Based on this, the present invention dynamically adjusts the scope of identity information obscured in the logs according to the size of the user group, ensuring that privacy is protected while accurately tracking user behavior across sources.

[0033] Specifically, multiple log data points are collected from various log sources. Each log data point includes a mask parameter, the number of active users within the current time window, masked identity information after processing user identity information using the mask parameter, user operation events, and their operation timestamps. The user identity information includes the user's phone number, ID card number, or application account, etc. The mask parameter includes information about reserved mask bits, such as the number of reserved mask bits and their sequence numbers. Taking a user's phone number as an example, assuming the reserved mask bits are 7 bits and the reserved mask bits are numbers 1-3 and 8-11, the generated masked identity information would be 123. The format is 4567.

[0034] These log data undergo preprocessing operations such as cleaning and correction, and are labeled with their corresponding source identifiers to form a log dataset with a uniform format.

[0035] Furthermore, all the log data is comprehensively analyzed. Based on the number of active users in each time window, the log data is divided into several user groups according to user size. For example, during peak hours, the number of active users is larger, and they are assigned to a high-density group; during off-peak hours, the number of active users is smaller, and they are assigned to a low-density group. For each user group, the masked identity information of each user in the group is compared to evaluate the degree of distinguishability between different users in the group, and a distinguishability score is obtained. The higher the distinguishability score, the greater the distinguishability of users in the group, and vice versa. For example, during peak hours, the number of active users is larger, and the corresponding high-density group usually has a lower distinguishability score, while during off-peak hours, the number of active users is smaller, and the corresponding low-density group usually has a higher distinguishability score.

[0036] After calculating the discrimination score for each user group, user groups with discrimination scores below a preset first score threshold are designated as low-discrimination groups, indicating a low degree of differentiation in masked identity information among users within that score range. For these low-discrimination groups, a target mask parameter different from the original mask parameter is generated to improve their discrimination score. Control commands are sent to the mask calculation units corresponding to each log source, carrying the target mask parameter and relevant information about the low-discrimination group, so that the mask calculation units reprocess the user identity information in the low-discrimination group using the target mask parameter to update the masked identity information of the users in the low-discrimination group before outputting it. Taking the aforementioned user phone number as an example, assuming the adjusted mask digits become 9, and the mask digits are numbered 1-5 and 8-11, the generated masked identity information is 12389. The format is 4567. Meanwhile, the masked identity information for users in other user groups remains unchanged.

[0037] Finally, based on the updated masked identity information in the log data, user operation events of the same user in different log sources are associated to form a user operation sequence. Combined with the corresponding source identifier, masked identity information, and other information, a user behavior dataset is obtained, which is then used for subsequent fault cause analysis.

[0038] By employing the technical means of this invention, multi-source log data is integrated, and a clustering algorithm is used to group users by size and evaluate the distinguishability of users within each group. The effectiveness of the mask configuration under the current user size is perceived in real time, and mask parameters are dynamically adjusted accordingly. For low-distinguishability groups, mask parameters are dynamically adjusted to enhance the distinguishability, avoiding both excessive masking that makes user identities indistinguishable and insufficient masking that leads to privacy leaks. This achieves a dynamic balance between privacy protection strength and fault tracing accuracy. By uniformly processing multi-source logs with the adjusted mask parameters and combining them with time-series aggregation, operation records of the same user scattered across different log sources can be accurately merged while protecting privacy, thus reconstructing the user's operation trajectory. This overcomes the problem of incompatibility between internal identifiers of different log sources, which hinders correlation and facilitates fault root cause tracing, improving the accuracy of fault cause tracing.

[0039] As a preferred embodiment, the present invention further implements the above embodiments, and after collecting log data from several log sources, the method further includes: All the log data are time-synchronized and source-identified to generate a log dataset in a uniform format; The time synchronization process includes: Using a preset standard time base as a reference, calculate the offset between the operation timestamp corresponding to the user operation event of each log data and the standard time base; The timestamp of each operation is compensated and corrected according to the offset to obtain the corrected log record.

[0040] In this embodiment of the invention, a log collection module connects to multiple log sources, including application server logs, gateway logs, and database operation logs. Mask parameters, the number of currently active users, user operation events, and user operation timestamps are obtained from each log source. Fault event logs can also be collected. The mask length, number of active users, user operation events, user operation timestamps, and fault event logs are categorized and labeled according to source identifiers to obtain raw log data with source identifiers. This source identifier field is used to distinguish the original location where the log data was generated.

[0041] Optionally, the source identifier is marked using a preset encoding rule. Different log sources correspond to different encoding values. The log collection module writes the encoding value into the source identifier field of each log record to form an original log record with a source identifier.

[0042] Furthermore, based on the operation timestamps of each log source in the original log data, and with a preset standard time base as a reference, the offset between each source timestamp and the standard time base is calculated. After compensating and correcting the offset, the corrected original log records are arranged and combined according to a unified time format and field format to obtain a unified format log dataset.

[0043] In one embodiment, for the operation timestamps from each source in the original log records, the log acquisition module pre-sets a standard time base, which serves as the reference point for all source timestamps. The log acquisition module reads the operation timestamps from each source, calculates the difference between each timestamp and the standard time base as an offset, and performs addition or subtraction compensation on the original timestamps according to the offset to unify the corrected timestamps under the same time base. Then, the corrected original log records are arranged according to a preset field order to obtain a log dataset with a unified format.

[0044] By employing the technical means of this invention, the offset between the timestamp of each log source and the standard time base is calculated and compensated, thus eliminating the time deviation caused by clock asynchrony between different systems. This ensures the accuracy of the time dimension during subsequent cross-source log time-series aggregation and avoids erroneous merging or omissions due to time deviation.

[0045] As a preferred embodiment, the present invention further implements the above embodiments. Step S12, namely, grouping the log data according to user scale based on the number of active users and the mask identity information to generate several user groups, and evaluating the distinguishability between different users in each user group to obtain a distinguishability score, includes steps S121 to S124: S121. Associate the number of active users with the masked identity information to generate a user scale feature vector; S122. Based on the user scale feature vector, the log data is divided into user groups of different user scales using a preset clustering algorithm; S123. Compare the masked identity information of different users in each user group, and count the user pairs that cannot be effectively distinguished by the masked identity information, and record them as similar user pairs. S124. Determine the discrimination score of the user group based on the proportion of the number of similar user pairs to the total number of user pairs in the user group.

[0046] In this embodiment of the invention, a mask retention bit field, a mask retention bit content field, and an active user count field are extracted from a unified format log dataset. The mask retention bit field records the number of bits retained when masking user identity information, and the active user count field records the total number of users currently performing operations within the current time window. The number of bits currently retained in the mask is obtained based on the mask retention bit field, and the total number of users within the current time window is obtained based on the active user count field. The mask retention bit field and the total number of users are then associated and paired to form a tuple, resulting in a user scale feature vector. This feature vector reflects the correspondence between mask configuration and user scale and is used for subsequent clustering.

[0047] Based on the user scale feature vector, the K-means clustering algorithm is used to group the user scale. The user scale feature vector is used as the clustering feature, and the user is divided into different clusters according to the density of users. Each cluster corresponds to a user scale group, resulting in a set of user scale groups.

[0048] Specifically, the application process of the K-means clustering algorithm is as follows: Using user-scale feature vectors as input data, the number of clusters is pre-defined. Based on the distribution range of the total number of users, the feature vectors are divided into different clusters. During clustering, the algorithm calculates the distance from each feature vector to the cluster center and assigns the feature vector to the cluster containing the nearest cluster center. This process is iterated until the cluster centers no longer change. Each cluster corresponds to a user-scale group; groups with high user density contain more active users, while groups with low user density contain fewer active users, thus obtaining a set of user-scale groups.

[0049] Taking a real-world operation and maintenance scenario as an example, each record in a unified format log dataset contains:

[0050] The system extracts the number of mask bits (denoted as m) from these records, representing the number of bits to be retained in the user identity field as configured by the current system, and the number of active users (denoted as n), representing the total number of users in the current time window. These two fields are combined into a two-dimensional feature vector: (m, n), which is used as the input for grouping.

[0051] The required number of bits for the mask varies depending on the user size. When n is small, 7 bits may be sufficient to distinguish all users; however, when n is large, 7 bits may be far from enough, resulting in many users having the same identity after masking. Therefore, the goal of grouping is to group time periods with similar user sizes together and evaluate whether their distinguishability is sufficient under the current mask configuration.

[0052] For example, suppose a ticketing platform collects log data for six time windows within a day, and the number of active users and the current mask for each window are configured as follows:

[0053] The current mask configuration is 7 digits (e.g., a mobile phone number is displayed as 138). (5678 format), all windows have the same mask bit length (m=7), but the number of users n varies greatly.

[0054] The system uses the K-means clustering algorithm to group these 6 feature vectors. The number of clusters is set to K=2, meaning there are two groups: a high-density group and a low-density group. In practical applications, the value of K can be adjusted according to the specific scenario.

[0055] Two cluster centers are randomly initialized. Assume the initial center points are selected as follows: Cluster 1 center: (7, 100), representing low density; Cluster 2 center: (7, 7000), representing high density.

[0056] Calculate the distance from each eigenvector to the centers of the two clusters using the Euclidean distance formula (the straight-line distance in the eigenvector space). For example, the eigenvector (7, 8000) has the following distances: to the center of cluster 1 (7, 100) = sqrt((7-7)² + (8000-100)²) = sqrt(0 + 7900²) = 7900; and to the center of cluster 2 (7, 7000) = sqrt((7-7)² + (8000-7000)²) = sqrt(0 + 1000²) = 1000. Since 1000 < 7900, (7, 8000) is assigned to cluster 2.

[0057] After assigning all feature vectors to the nearest cluster, recalculate the mean of all points within each cluster, and use this mean as the new cluster center. Repeat this process until the cluster centers no longer change (convergence).

[0058] After K-Means clustering, the six time windows were divided into two groups:

[0059] Each cluster corresponds to a user size group, reflecting the differences in user density over different time periods.

[0060] Further, the discrimination score of each user group is calculated.

[0061] Preferably, the mask parameter includes information on mask reserved bits; the step of comparing the mask identity information of different users within each user group, and statistically analyzing user pairs that cannot be effectively distinguished by the mask identity information, and recording them as similar user pairs, includes: Form a user pair from any two users within the same user group; Compare the values ​​of the mask reserved bits in the mask identity information of the user pair bit by bit, and count the number of bits with the same value, which are recorded as the overlapping bits; When the number of overlapping bits exceeds a preset overlap threshold, the user pair is determined to be a similar user pair that cannot be effectively distinguished.

[0062] In this embodiment of the invention, for each group in the user scale grouping set, the mask reserved bit content of each user in the group is compared one by one, and the number of bits with the same mask reserved bit content between any two users is counted as the overlap number. If the overlap number exceeds a preset overlap threshold, it is determined that the two users are difficult to distinguish. Based on the proportion of the number of difficult-to-distinguish user pairs to the total number of user pairs in the group, the user discrimination score of the group is determined. The user discrimination scores of each group are summarized and marked to obtain the user group discrimination evaluation result.

[0063] The discrimination score is determined as follows: The number of indistinguishable user pairs within a group is counted, and the proportion of this number to the total number of user pairs within the group is calculated. A higher proportion indicates lower discrimination, and a lower proportion indicates higher discrimination. The reciprocal or complement of this proportion is used as the discrimination score. The discrimination scores of each group are summarized and labeled to form the user group discrimination evaluation result. This evaluation result can intuitively reflect the user identity discrimination capability of each user size group under the current mask configuration.

[0064] In the example above, after grouping, the system calculates a discrimination score for each group separately. Assuming that only two pairs of users in the low-density group are indistinguishable because the first three and last four digits of their phone numbers are identical, the calculated discrimination score is 0.9, higher than the preset score threshold of 0.7, therefore no adjustment is triggered. However, in the high-density group, a large number of users' phone numbers are indistinguishable because the first three and last four digits are identical, resulting in a discrimination score of 0.65, lower than the score threshold, thus triggering an adjustment.

[0065] By employing the technical means of this invention, a user scale feature vector is generated by associating the number of bits retained in the mask with the total number of users, and then grouping them using a clustering algorithm. This allows scenarios with similar user scales to be grouped into the same group for differentiated evaluation. Furthermore, by statistically analyzing the proportion of users whose identity information cannot be effectively distinguished after masking within a group, a quantitative method for evaluating the effectiveness of mask configuration is provided. This provides objective data for subsequent mask adjustments and avoids the subjectivity and blindness of setting mask parameters based on experience.

[0066] As a preferred embodiment, the present invention further implements the above embodiments. Step S13, namely, processing the user identity information in the user group whose discrimination score is lower than a preset first score threshold using target mask parameters to update the masked identity information, includes steps S131 to S133: S131. Obtain user groups whose discrimination scores are lower than a preset first score threshold, and denot them as target user groups; S132. Generate target mask parameters based on the total number of users in the target user group and the original mask parameters; S133. The target mask parameters are used to process the user identity information in the target user group to update the mask identity information of the user in the target user group.

[0067] Preferably, the mask parameters include the number of mask bits to retain; generating target mask parameters based on the total number of users in the target user group and the current mask parameters includes: Obtain the total number of users in the target user group, and calculate the theoretical number of bits required to distinguish this number of users based on the total number of users; Based on the difference between the theoretical number of bits to be retained and the current number of bits to be retained, the increment of the number of bits to be retained is determined, and the increment is added to the current number of bits to be retained to obtain the adjusted target number of bits to be retained.

[0068] The formula for calculating the increment is:

[0069] in, The increment is n, where n is the total number of users. To retain the theoretical number of bits, Reserve bits for the current mask.

[0070] In this embodiment of the invention, the user discrimination score of each group is read one by one from the user group discrimination evaluation results, and the user discrimination score is compared with a preset discrimination threshold. If the user discrimination score of a group is lower than the discrimination threshold, it is determined that the group cannot effectively distinguish the identities of different users under the current mask configuration, and the group is marked as a low discrimination level group and added to the low discrimination group set.

[0071] The mask calculation unit is the core processing module for adjusting the reserved bit depth. Its working principle is as follows: For each group in the low-discrimination group set, the current reserved bit depth of the mask for that group and the total number of users within that group are obtained. The mask calculation unit calculates the minimum number of bits theoretically needed to distinguish that number of users based on the logarithm of the total number of users. This logarithm is compared with the current reserved bit depth; the difference between the two is the increment value for the reserved bit depth. This increment value is then added to the current reserved bit depth to obtain the adjusted reserved bit depth for that group.

[0072] It should be noted that the reason for using logarithmic values ​​as the basis for calculation is that there is an exponential relationship between the number of bits reserved in the mask and the number of distinguishable users. For every additional bit of reserved bits, the number of distinguishable users doubles.

[0073] For example, in the log processing scenario of the operation and maintenance platform, there are 1,000 active users in a certain group. The current mask retains 8 bits. The base-2 logarithm of 1,000 is approximately 10 bits. Therefore, the increment value is 2 bits, and the adjusted mask retains 10 bits.

[0074] Optionally, the adjusted mask retention bits of each group in the low-discrimination group set are summarized, and the maximum value of the adjusted retention bits of all groups is taken as the global unified mask parameter. The parameters are recorded according to the correspondence between the group identifier and the adjusted mask retention bits to obtain a mask parameter set with enhanced discrimination.

[0075] After the mask calculation unit completes the adjustment of the reserved number of bits for all low-discrimination groups, it establishes a correspondence between the group identifier of each group and the adjusted reserved number of bits in the mask, and summarizes them to form a mask parameter set that enhances discrimination. This mask parameter set records the adjusted mask configuration information of each low-discrimination group, enabling more identity discrimination bits to be retained in high-user-density scenarios.

[0076] By employing the technical means of this invention, for target groups with insufficient distinguishability, the minimum number of bits theoretically required to distinguish all users is calculated based on the total number of users within the group. This calculation then determines the increment of the mask's reserved bits, ensuring that the mask adjustment range precisely matches the user scale. By processing log records for groups with different user scales using corresponding mask parameters, differentiated configuration of the mask granularity is achieved. This on-demand adjustment method avoids over-adjustment or under-adjustment caused by fixed increments, ensuring the adaptability of mask parameters across different user scale scenarios.

[0077] As a preferred embodiment, the present invention further implements the above embodiments. Step S14, namely, associating user operation events of the same user in different log sources to form a user operation sequence based on the updated mask identity information to obtain a user behavior dataset, includes steps S141 to S144: S141. Using the time window as the classification dimension, aggregate log data from different log sources within the same time window; S142. User operation events that are within the same time window and have the same mask identity information are grouped into user operation events of the same user. S143. Sort the user operation events of the same user according to the corresponding operation timestamp to form the user operation sequence of the user; S144. The user operation sequence is associated with and stored with the corresponding mask identity information and the source identifier of each user operation event to obtain a user behavior dataset.

[0078] In this embodiment of the invention, the adjusted mask retention bits corresponding to each group are obtained from the mask parameter set for enhanced discrimination. The mask parameter set for enhanced discrimination is pre-constructed based on the source identifier of each log record in the unified format log dataset. This set is generated by calculating the optimal mask retention bits for each group based on user size grouping. The adjusted mask retention bits corresponding to each group are obtained from the mask parameter set for enhanced discrimination. Based on the source identifier of each log record in the unified format log dataset, the user size group to which the log record belongs is matched. The user identity field in the log record is masked according to the mask retention bits corresponding to the group, retaining a specified number of identity information bits and hiding the remaining bits, thus obtaining the cross-source log record after dynamic masking.

[0079] Based on the user operation timestamps in the cross-source log records after dynamic masking, the log records are divided according to a preset time window length. Within the same time window, log records with the same identity information retained after masking are identified as operations by the same user. These operation records of the same user are merged to obtain a user operation sequence divided by time window. For the user operation sequence, the log records under the same user are arranged according to the order of operation timestamps. The arranged operation records are associated and stored with the corresponding source identifier and the masked identity information to obtain a user behavior aggregated dataset.

[0080] Specifically, in one implementation, the adjusted mask retention bits corresponding to each user size group are read from the mask parameter set that enhances distinguishability. Based on the source identifier carried by each log record in the unified format log dataset, the user size group to which the log record belongs is found, and the mask retention bits corresponding to the group are obtained.

[0081] The dynamic masking process is as follows: For the user identity field in the log record, according to the obtained mask retention bit length, a specified number of bits are retained starting from the beginning of the user identity field, and the remaining bits are replaced with a preset mask symbol. User size grouping is based on the number of users. If the number of users is greater than 1000, it is a high user density group; if the number of users is less than or equal to 1000, it is a low user density group. Different groups of log records use different retention bit lengths for masking. High user density groups retain more bits to enhance distinguishability, while low user density groups retain fewer bits to strengthen privacy protection. After masking, the dynamically masked cross-source log records are obtained.

[0082] It should be noted that the time window is divided based on the user's operation timestamp. According to the preset time window length, the dynamically masked cross-source log records are divided into corresponding time windows, with each time window containing log records from all sources within that time period.

[0083] In one embodiment, the method for determining the same user is as follows: within the same time window, compare the identity information retained after masking of each log record. If the identity information retained by two log records is exactly the same, then the two records are determined to belong to the same user's operation and are merged into the same user's operation sequence.

[0084] For example, in an operations and maintenance platform scenario, two records from application server logs and gateway logs, if they occur within the same time window and the identity information retained after masking is consistent, and the IP address or session ID matches, are merged into an operation sequence of the same user. Further, for the merged user operation sequence, the log records are arranged in ascending order of operation timestamps. The arranged operation records are then associated with their corresponding source identifiers and masked identity information and stored to form a user behavior aggregated dataset. This dataset records the operation trajectory information of each user from different sources.

[0085] By employing the technical means of this invention, through time window division and matching and merging of mask identity information, the operation records of the same user in multiple log sources are linked into a complete operation sequence, providing a structured data foundation for subsequent user behavior analysis and fault root cause tracing, and solving the problem of difficulty in associating multi-source heterogeneous logs.

[0086] For a preferred embodiment, see Figure 2 This is a flowchart illustrating a preferred log data processing method in an embodiment of the present invention. The embodiments of the present invention are further implemented based on any of the above embodiments, and the log data also includes fault events. After associating user operation events of the same user in different log sources based on the updated mask identity information to obtain a user behavior dataset, the method further includes step S15: S15. Trace the cause of the fault based on the user behavior dataset and the fault event.

[0087] In this embodiment of the invention, fault event logs are extracted from a unified format log dataset, and the user behavior dataset and the fault event logs are matched to analyze the cause of the fault.

[0088] In a preferred embodiment, before tracing the cause of the fault based on the user behavior dataset and the fault event, the method further includes: Based on the user behavior dataset, the user's behavioral features are extracted, and a preset classification algorithm is used to distinguish the user into real users and fake users, resulting in user operation sequences with identity labels.

[0089] Preferably, the behavioral characteristics include at least one of operation frequency, operation time interval, number of cross-source switching, and operation type distribution; The step of extracting user behavior features from the user behavior dataset and using a preset classification algorithm to distinguish users into real users and fake users to obtain user operation sequences with identity labels includes: Extract the behavioral features of each user within the time window from the user behavior dataset; the behavioral features include one or more of the following: operation frequency, operation time interval, number of cross-source switching, and operation type distribution. The aforementioned behavioral features are combined into a user behavior feature vector; A pre-built classification model is used to distinguish the user behavior feature vectors and output user categories, which include real users and fake users; Based on the discrimination results, the corresponding user operation sequences are labeled with identities to obtain user operation sequences with identity labels.

[0090] Preferably, the preset classification algorithm is a random forest model based on decision tree ensemble, which is constructed using historical behavioral feature samples labeled with real users or fake users as training data.

[0091] In this embodiment of the invention, the operation records of each user within a time window are read from the user behavior aggregated dataset, and behavioral features are extracted for each user's operation records. The behavioral features include operation frequency, operation time interval, number of cross-source switching and operation type distribution, and the behavioral features are combined into a user behavior feature vector.

[0092] Specifically, in one implementation, the operation records of each user within a time window are read from the user behavior aggregated dataset, and behavioral features are extracted for each user's operation record. The extraction process of behavioral features includes: counting the total number of operations performed by the user within the time window and dividing by the time window length to obtain the operation frequency; calculating the time difference between two adjacent operations and taking the average to obtain the operation time interval; counting the number of times the source identifier changes in the user's operation records to obtain the number of cross-source switching; and counting the proportion of each type of operation performed by the user to obtain the operation type distribution. The above four behavioral features are combined to form a user behavior feature vector.

[0093] Furthermore, based on the user behavior feature vectors, a random forest algorithm is used to classify users. This algorithm uses a pre-constructed training sample set as training data, which includes labeled features of both fake and real user behavior. The algorithm discriminates the behavior feature vectors of users to be classified, outputting a classification result indicating whether the user is a fake or real user. Based on the classification result, each user operation sequence in the user behavior aggregate dataset is labeled with an identity. Operation sequences classified as belonging to real users are labeled with a real user identifier, and operation sequences classified as belonging to fake users are labeled with a fake user identifier, resulting in a set of user operation sequences with identity labels.

[0094] Specifically, the Random Forest algorithm is a classification method based on decision tree ensembles. Its working principle is as follows: Several subsets of samples are randomly selected from the training sample set. A decision tree is constructed for each subset. Each decision tree independently judges the data to be classified and outputs a classification result. Finally, the classification results of multiple decision trees are aggregated through a voting mechanism, and the category with the most votes is the final classification result. In this scheme, the Random Forest algorithm uses user behavior feature vectors as input and fake users or real users as the output category.

[0095] The training sample set is constructed as follows: user operation records with known identities are selected from historical log data, their behavioral features are extracted, and they are labeled as real user samples; simultaneously, user operation records with spoofed behavior are selected from identified abnormal operation records or security event records, their behavioral features are extracted, and they are labeled as spoofed user samples. The real user samples and spoofed user samples are then merged to form the training sample set.

[0096] For example, in the log processing scenario of an operations and maintenance platform, spoofed users typically exhibit behavioral patterns different from real users: their operation frequency is often abnormally high (e.g., more than 10 times per minute) or low (less than 1 time per minute), the operation time intervals show an irregular distribution (standard deviation greater than 5 seconds), the number of cross-source switching is significantly higher than normal users (more than 5 times / hour), and the operation types are concentrated on specific sensitive operations (concentration greater than 80%). The Random Forest algorithm learns these behavioral differences and distinguishes the behavioral feature vectors (with a dimension of n, where n represents the number of features) of the users to be classified, achieving a classification accuracy of over 95%.

[0097] Understandably, the purpose of identity labeling is to distinguish user behavior patterns of different natures, so as to eliminate interference from fake users and focus on the operation path of real users in the subsequent root cause tracing process.

[0098] Optionally, the set of user operation sequences with identity annotations is organized according to time windows, retaining each user's operation timestamp, source identifier, masked identity information, and identity annotation results. These are then arranged chronologically to form a complete user operation trajectory, resulting in a differentiated user identity trajectory. This user identity trajectory records each user's operational behavior and identity attributes within different time windows, supporting subsequent fault root cause localization and tracing.

[0099] By employing the technical means of this invention, multi-dimensional behavioral features are extracted to comprehensively depict the user's operational behavior patterns. A classification model is used to automatically distinguish between real users and fake users. Compared with manual rule judgment, this method has higher accuracy and efficiency, and can effectively identify the behavioral trajectories of fake users such as crawlers and attack programs, thus avoiding their interference with the root cause analysis of faults.

[0100] As a preferred embodiment, the present invention further implements the above embodiments. Step S15, namely, tracing the cause of the fault based on the user behavior dataset and the fault event, includes steps S151 to S155: S151. Determine the fault time window based on the fault occurrence time in the fault event; S152. Based on the user operation sequence in the user behavior dataset, filter the target operation sequence whose operation timestamp falls within the fault time window; S153. Compare the operation timestamps and source identifiers of each user operation event in the target operation sequence with the fault trigger time and fault source in the fault event, respectively. S154. Calculate the matching score based on the proportion of the number of events that match to the total number of events in the target operation sequence; S155. Generate the fault triggering reason based on the target operation sequence whose matching score exceeds the preset second score threshold.

[0101] Preferably, the step of filtering target operation sequences whose operation timestamps fall within the fault time window based on the user operation sequences in the user behavior dataset specifically involves: Filter user action sequences labeled as real users from the user behavior dataset; From the user operation sequences of the real users, filter the target operation sequences whose operation timestamps fall within the fault time window.

[0102] In this embodiment of the invention, operation sequences labeled with real user identifiers are filtered from the differentiated user identity trajectories. Fault event logs are extracted from a unified format log dataset. A fault time window is determined based on the fault occurrence timestamp in the fault event log. Operation records with time overlap with the fault time window are searched in the operation sequences of the real user identifiers to obtain a set of operation sequences to be matched. For each operation sequence in the set of operation sequences to be matched, the operation timestamp and source identifier of each operation record are compared one by one with the fault trigger time and fault source in the fault event log. The number of operation records with timestamp differences within a preset time tolerance range and consistent source identifiers is counted. A matching degree score is calculated based on the proportion of the number of matched operation records to the total number of records in the operation sequence to obtain the matching degree score of each operation sequence. The matching degree score is compared with a preset matching level threshold. If the matching degree score exceeds the matching level threshold, the operation sequence is marked as high matching level; operation sequences that do not exceed the threshold are marked as low matching level, resulting in a set of operation sequences with matching level labels. The operation sequences with high matching levels are extracted from the set of operation sequences with matching level markers. The user operation path is restored according to the order of operation timestamps as the root cause path. The root cause path is associated with the corresponding masked identity information and matching degree score. The proportion of the number of operation sequences with high matching levels to the total number of operation sequences is used as the tracing success rate, and the root cause tracing record is obtained.

[0103] Specifically, in one implementation, operation sequences labeled as genuine user identifiers are filtered from the differentiated user identity trajectories, while operation sequences labeled as fake user identifiers are excluded. Simultaneously, fault event logs are extracted from a unified format log dataset. These fault event logs record the timestamp of the fault occurrence, the source of the fault, and the type of the fault. Based on the fault occurrence timestamp in the fault event logs, preset time ranges are extended forward and backward to form a fault time window.

[0104] The process for filtering operation sequences to be matched is as follows: Iterate through all operation sequences identified by real users, checking if any operation record in each sequence has an operation timestamp falling within the fault time window. If at least one operation record's timestamp overlaps with the fault time window, then that operation sequence is included in the set of operation sequences to be matched. This filtering method narrows the matching range, focusing on active user operations before and after the fault occurred.

[0105] It's important to note that comparing the timestamp and the source identifier is the core step in calculating the match rate. For each operation sequence in the set of operation sequences to be matched, the operation records are read one by one. The difference between the operation timestamp of each operation record and the fault trigger time in the fault event log is calculated, and it is determined whether the difference is within the preset time tolerance range. Simultaneously, the source identifier of the operation record is matched against the fault source in the fault event log to determine if they are identical. Only when the timestamp difference is within the tolerance range and the source identifier matches is the operation record considered a successful match.

[0106] For example, in an operations and maintenance platform scenario, if the fault trigger time recorded in the fault event log is a certain moment, and the time tolerance is set to a certain number of seconds before and after, then operation records whose operation timestamp and fault trigger time are within this range are considered time matches. If the source identifier of the operation record is a gateway log, and the fault source is also the gateway, then the source match is successful.

[0107] In one embodiment, the matching score is calculated as follows: the number of successfully matched operation records in the operation sequence is counted, and then divided by the total number of operation records in the operation sequence to obtain the matching score. The matching score ranges from zero to one; a higher value indicates a stronger correlation between the user's operation and the fault event. Further, a level labeling is performed based on the matching score. The matching score of each operation sequence is compared with a preset matching level threshold. If the matching score exceeds the threshold, the operation sequence is labeled as a high matching level; otherwise, it is labeled as a low matching level. Operation sequences with high matching levels are considered to have a strong correlation with the fault event and are key targets for root cause tracing.

[0108] Understandably, the root cause path reconstruction process involves transforming high-match-level operation sequences into traceable operation paths. All high-match-level operation sequences are extracted from the set of operation sequences marked with match levels. These sequences are then arranged in ascending order of their timestamps to form the complete operation path from the start of the user's action to the triggering of the fault—the root cause path. This root cause path reveals the user's operational trajectory before the fault occurred, facilitating the identification of the specific action that caused the fault.

[0109] In one possible implementation, the root cause tracing record comprises three parts: the root cause path, masked identity information, and the tracing success rate. The root cause path records the operation trajectory of users with high matching levels; the masked identity information retains some user identity features to support subsequent analysis while hiding sensitive information to protect privacy; the tracing success rate is the ratio of the number of high-matching operation sequences to the total number of operation sequences to be matched, reflecting the overall effectiveness of this fault tracing. This root cause tracing record can provide accurate location data for the root cause of a fault while protecting user privacy.

[0110] By employing the technical means of this invention, trajectories that match the fault time window are filtered from the user operation sequence. The matching degree score is calculated by comparing the timestamp and the source identifier, and the score is used to mark the level. This allows for the rapid and objective filtering of the operation path with the highest correlation to the fault event from a large number of user trajectories, which greatly improves the efficiency and accuracy of fault root cause tracing.

[0111] In a preferred embodiment, after step S15, the method further includes: Target operation sequences whose matching scores exceed a preset second score threshold are marked as high matching levels; The ratio of the number of user operation sequences with high matching levels to the number of target operation sequences is calculated as the tracing success rate. The mask parameters are adjusted based on the comparison between the traceability success rate and the preset success rate threshold.

[0112] In this embodiment of the invention, the adjusted mask parameters are used for subsequent log masking processing and user behavior aggregation.

[0113] Preferably, the mask parameters are adjusted based on a comparison between the traceability success rate and a preset success rate threshold, including: The traceability success rate is compared with a preset success rate threshold. When the tracing success rate is lower than the preset success rate threshold, the user scale group to which the source identifier belongs is determined based on the source identifier corresponding to the root cause path, and is designated as the group to be adjusted. For the group to be adjusted, a preset bit increment is added to the current mask retention bit depth to obtain an optimized mask retention bit depth, and the mask parameter set is updated.

[0114] In this embodiment of the invention, the tracing success rate is read from the root cause tracing record. This success rate is compared with a preset success rate threshold. If the success rate is lower than the threshold, the user size group to which the source identifier belongs is found based on the source identifier corresponding to the root cause path in the root cause tracing record. This user size group is marked as a group to be adjusted, resulting in a set of group identifiers to be adjusted. For each group in the set of group identifiers to be adjusted, the current mask retention bit length is obtained from the mask parameter set for enhanced differentiation. A preset bit increment is added to the current retention bit length using a mask calculation unit to obtain the optimized mask retention bit length for the group. The optimized mask retention bit length replaces the original retention bit length, resulting in an updated mask parameter set. The updated mask parameter set is applied to subsequent cross-source log masking and user behavior aggregation processes. Dynamic masking and time-series aggregation are performed on newly collected log data according to the optimized mask retention bit length, continuously performing user identity differentiation and fault root cause tracing.

[0115] Specifically, in one implementation, the feedback loop mechanism is a processing method that dynamically adjusts the mask configuration based on the tracing results. The tracing success rate is read from the root cause tracing record. This success rate reflects the proportion of high-match-level operation sequences to the total number of operation sequences, where high-match-level operation sequences refer to operation sequences with a match degree exceeding a preset threshold M (e.g., M=0.8). The tracing success rate is compared with a preset success rate threshold. If the tracing success rate is lower than the success rate threshold, it indicates that the user discrimination under the current mask configuration is insufficient to support accurate fault tracing.

[0116] The process for identifying groups to be adjusted is as follows: Root cause paths are extracted from the root cause tracing records. The source identifiers of each operation record within the root cause path are read. Based on these source identifiers, the corresponding log records are searched in a unified format log dataset to obtain the user-scale group to which the log record belongs. User-scale groups with tracing success rates below a threshold are marked as groups to be adjusted, forming a set of group identifiers to be adjusted. This method allows for precise identification of which groups have inadequate mask configurations.

[0117] It should be noted that the mask parameter set for enhancing distinguishability is a pre-generated set of parameters used to improve the distinguishability of groups, including the initial mask reserved number of bits for each group. When the mask calculation unit performs reserved number of bits adjustment, for each group in the set of group identifiers to be adjusted, it obtains the current mask reserved number of bits for that group from the set, and adds a preset bit increment to the current reserved number of bits to obtain the optimized mask reserved number of bits.

[0118] For example, in an operations and maintenance platform scenario, if the current IP address mask reserved bit length for a certain user group is 10 bits and the preset bit length increment is 2 bits, then the optimized mask reserved bit length is 12 bits.

[0119] In one embodiment, the original reserved bit length is replaced with the optimized reserved bit length for all groups to be adjusted, forming an updated mask parameter set. The updated mask parameter set is applied to subsequent cross-source log masking and user behavior aggregation processes. Newly collected log data is dynamically masked according to the optimized reserved bit length, continuously distinguishing user identities and tracing root causes of faults, forming a closed-loop mask configuration optimization process.

[0120] By employing the technical means of this invention, the traceability success rate is used as a feedback signal. When the success rate is lower than the threshold, the mask parameter is automatically optimized, forming a closed-loop iterative mechanism. This enables the system to continuously improve itself based on the actual traceability effect, maintain stable traceability performance in constantly changing business scenarios, and eliminate the need for frequent manual intervention.

[0121] See Figure 3 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present invention. The present invention also provides a log data processing device 10, comprising: The log data acquisition module 11 is used to collect log data from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; The discrimination score calculation module 12 is used to group the log data according to the user scale based on the number of active users and the mask identity information, generate several user groups, and evaluate the degree of distinguishability between different users in each user group to obtain a discrimination score. The mask parameter adjustment module 13 is used to process the user identity information in the user group whose discrimination score is lower than a preset first score threshold using a target mask parameter to update the mask identity information; wherein, the target mask parameter is used to improve the discrimination score; The user behavior association module 14 is used to associate user operation events of the same user in different log sources to form a user operation sequence based on the updated mask identity information, thereby obtaining a user behavior dataset.

[0122] In a preferred embodiment, the log data also includes fault events.

[0123] The device further includes: The fault cause tracing module is used to trace the fault cause based on the user behavior dataset and the fault event.

[0124] It should be noted that the log data processing device provided in this embodiment of the invention is used to execute all the process steps of the log data processing method in the above embodiment. The working principle and beneficial effects of the two are one-to-one, so they will not be described again.

[0125] See Figure 4 This is a schematic diagram of the structure of a log data processing device provided in an embodiment of the present invention. The present invention also provides a log data processing device 20, including a processor 21, a memory 22, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the log data processing method as described in any of the above embodiments.

[0126] This invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the log data processing method as described in any of the above embodiments.

[0127] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the log data processing method as described in any of the above embodiments.

[0128] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0129] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A log data processing method, characterized in that, include: Log data is collected from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; Based on the number of active users and the masked identity information, the log data is grouped according to user scale to generate several user groups, and the degree of distinguishability between different users in each user group is evaluated to obtain a distinguishability score. For user identity information in user groups whose discrimination scores are lower than a preset first score threshold, a target mask parameter is used to process the masked identity information to update the masked identity information; wherein, the target mask parameter is used to improve the discrimination score; Based on the updated mask identity information, user operation events of the same user in different log sources are associated to form a user operation sequence, thus obtaining a user behavior dataset.

2. The log data processing method as described in claim 1, characterized in that, The process involves grouping the log data according to user size based on the number of active users and the masked identity information, generating several user groups, and evaluating the distinguishability between different users in each user group to obtain a distinguishability score, including: The number of active users and the masked identity information are correlated to generate a user scale feature vector; Based on the user size feature vector, a preset clustering algorithm is used to divide the log data into user groups of different user sizes; Compare the masked identity information of different users within each user group, and count the user pairs that cannot be effectively distinguished by the masked identity information, and record them as similar user pairs; The discrimination score of a user group is determined based on the proportion of the number of similar user pairs to the total number of user pairs within the user group.

3. The log data processing method as described in claim 2, characterized in that, The mask parameters include information about the mask reserved bits; The step of comparing the masked identity information of different users within each user group and statistically analyzing user pairs that cannot be effectively distinguished by the masked identity information is denoted as similar user pairs, including: Form a user pair from any two users within the same user group; Compare the values ​​of the mask reserved bits in the mask identity information of the user pair bit by bit, and count the number of bits with the same value, which are recorded as the overlapping bits; When the number of overlapping bits exceeds a preset overlap threshold, the user pair is determined to be a similar user pair that cannot be effectively distinguished.

4. The log data processing method as described in claim 1, characterized in that, The user identity information in user groups whose discrimination scores are lower than a preset first score threshold is processed using target mask parameters to update the masked identity information, including: The user group whose discrimination score is lower than a preset first score threshold is identified and recorded as the target user group. Generate target mask parameters based on the total number of users in the target user group and the original mask parameters; The target mask parameters are used to process the user identity information in the target user group in order to update the mask identity information of the users in the target user group.

5. The log data processing method as described in claim 4, characterized in that, The mask parameters include information about the mask reserved bits; The step of generating target mask parameters based on the total number of users in the target user group and the current mask parameters includes: Obtain the total number of users in the target user group, and calculate the theoretical number of bits required to distinguish this number of users based on the total number of users; Based on the difference between the theoretical number of bits to be retained and the current number of bits to be retained, the increment of the number of bits to be retained is determined, and the increment is added to the current number of bits to be retained to obtain the adjusted target number of bits to be retained.

6. The log data processing method as described in claim 1, characterized in that, The step involves associating user operation events of the same user in different log sources based on the updated mask identity information to form a user operation sequence, thereby obtaining a user behavior dataset, including: Using the time window as the classification dimension, log data from different log sources within the same time window are aggregated; User operation events that are within the same time window and have the same mask identity information are grouped into user operation events of the same user. The user operation events of the same user are sorted according to the corresponding operation timestamps to form the user operation sequence of the user; The user operation sequence is associated with and stored with the corresponding mask identity information and the source identifier of each user operation event to obtain a user behavior dataset.

7. The log data processing method as described in claim 1, characterized in that, After collecting log data from several log sources, the method further includes: All the log data are time-synchronized to generate a log dataset in a uniform format; The time synchronization process includes: Using a preset standard time base as a reference, calculate the offset between the operation timestamp corresponding to the user operation event of each log data and the standard time base; The timestamp of each operation is compensated and corrected according to the offset to obtain the corrected log record.

8. The log data processing method according to any one of claims 1 to 7, characterized in that, The log data also includes fault events; After associating user operation events of the same user in different log sources based on the updated mask identity information to obtain a user behavior dataset, the method further includes: The cause of the failure is traced based on the user behavior dataset and the failure event.

9. The log data processing method as described in claim 8, characterized in that, The step of tracing the cause of a fault based on the user behavior dataset and the fault event includes: Determine the fault time window based on the fault occurrence time in the fault event; Based on the user operation sequences in the user behavior dataset, filter the target operation sequences whose operation timestamps fall within the fault time window; The operation timestamps and source identifiers of each user operation event in the target operation sequence are compared with the fault trigger time and fault source in the fault event, respectively. The matching score is calculated based on the proportion of the number of events that match to the total number of events in the target operation sequence. Based on the target operation sequence whose matching score exceeds a preset second score threshold, generate the fault triggering reason.

10. The log data processing method as described in claim 9, characterized in that, Before tracing the cause of the failure based on the user behavior dataset and the failure event, the method further includes: Based on the user behavior dataset, the user's behavioral features are extracted, and a preset classification algorithm is used to distinguish the user into real users and fake users, resulting in user operation sequences with identity labels. The step of filtering target operation sequences whose operation timestamps fall within the fault time window based on the user operation sequences in the user behavior dataset specifically involves: Filter user action sequences labeled as real users from the user behavior dataset; From the user operation sequences of the real users, filter the target operation sequences whose operation timestamps fall within the fault time window.

11. The log data processing method as described in claim 10, characterized in that, The behavioral characteristics include at least one of operation frequency, operation time interval, number of cross-source switching, and operation type distribution; And / or, the preset classification algorithm is a random forest model based on decision tree ensemble.

12. The log data processing method as described in claim 9, characterized in that, The method further includes: Target operation sequences whose matching scores exceed a preset second score threshold are marked as high matching levels; The ratio of the number of user operation sequences with high matching levels to the number of target operation sequences is calculated as the tracing success rate. The mask parameters are adjusted based on the comparison between the traceability success rate and the preset success rate threshold.

13. A log data processing device, characterized in that, include: The log data acquisition module is used to collect log data from several log sources; wherein each log data includes a mask parameter, the number of active users in the current time window, masked identity information after processing user identity information using the mask parameter, and user operation events; The discrimination score calculation module is used to group the log data according to the user scale based on the number of active users and the mask identity information, generate several user groups, and evaluate the degree of distinguishability between different users in each user group to obtain a discrimination score. The mask parameter adjustment module is used to process the user identity information in the user group whose discrimination score is lower than a preset first score threshold using a target mask parameter to update the mask identity information; wherein, the target mask parameter is used to improve the discrimination score; The user behavior association module is used to associate user operation events of the same user in different log sources to form a user operation sequence based on the updated mask identity information, thereby obtaining a user behavior dataset.

14. A log data processing device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the log data processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the log data processing method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the log data processing method as described in any one of claims 1 to 12.