Log management method and device, equipment and medium

Abstract

The embodiment of the invention provides a log management method and device, equipment and a medium. The method comprises the steps that query keywords for logs are obtained; determining an association keyword according to the query keyword; the association keyword is generated according to semantic association of the query keyword; determining a reverse index field and a balanced tree index field from at least one field contained in the association keyword; determining at least one candidate log according to a reverse index field, a reverse index corresponding to the reverse index field, a balance tree index field and a balance tree index corresponding to the balance tree index field; respectively determining content keywords of at least one candidate log; according to the content keyword and the association keyword, respectively determining a display score of the at least one candidate log; and in the at least one candidate log, taking the log with the display score greater than a preset score threshold as a target log, and outputting the target log to the user, thereby improving the accuracy and efficiency of log query.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log management method, device, equipment and medium. Background Art

[0002] In today's information technology landscape, logging is a crucial component for ensuring stable application operation and troubleshooting. Traditional log management relies on specific logging frameworks, which capture application-generated log information and store it in log files deployed on individual servers. For example, core service components like task scheduling systems, configuration management centers, real-time monitoring systems, and application release systems all have their log data distributed across the servers corresponding to their deployment environments.

[0003] However, with the continuous improvement of enterprise informatization and the surge in the number of applications, this decentralized log storage method has exposed its inherent drawbacks. When users need to analyze and diagnose the operating status of a specific application, they must manually log in to the corresponding server node and obtain information by executing specific command line tools or viewing log files. Traditional log query methods rely on precise keyword matching and lack a deep understanding and expansion of user query intent. When querying logs, users often need to manually enter multiple relevant keywords to ensure that they can find all relevant log entries. This query method is not only time-consuming but also prone to missing important information, resulting in inefficient fault diagnosis and problem solving. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a log management method, apparatus, device, and medium that overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above problems, an embodiment of the present invention discloses a log management method, which includes:

[0006] Get the query keywords for the log;

[0007] Determining an associative keyword based on the query keyword; the associative keyword is generated based on the semantic association of the query keyword;

[0008] Determine an inverted index field and a balanced tree index field from at least one field included in the associative keyword;

[0009] Determine at least one candidate log according to the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field;

[0010] respectively determining content keywords of the at least one candidate log;

[0011] determining a display score of the at least one candidate log according to the content keyword and the association keyword;

[0012] Among the at least one candidate log, the log having a presentation score greater than a preset score threshold is selected as a target log, and the target log is output to the user.

[0013] Optionally, determining the display score of the at least one candidate log based on the content keyword and the association keyword includes:

[0014] Determining a matching degree between the content keyword and the associated keyword;

[0015] A display score of the at least one candidate log is determined according to the content keyword and the matching degree.

[0016] Optionally, the log keyword includes a node identifier, where the node identifier is used to indicate a node that generates the log. Determining a display score of the at least one candidate log based on the content keyword and the matching degree includes:

[0017] Determine a first retrieval frequency of the node within a first preset time period; the first retrieval frequency is used to indicate the frequency at which logs of the node are retrieved within the first preset time period;

[0018] A display score of the at least one candidate log is determined according to the content keyword, the matching degree, and the first retrieval frequency.

[0019] Optionally, determining a display score of the at least one candidate log according to the content keyword, the matching degree, and the first access frequency includes:

[0020] Determining a second access frequency of the log within a second preset time period; the second access frequency is used to indicate the number of times the log is accessed within the second preset time period;

[0021] A display score of the at least one candidate log is determined according to the second retrieval frequency, the matching degree, and the first retrieval frequency.

[0022] Optionally, the method further includes:

[0023] After outputting the target log to the user, using the target log as a monitoring log to be monitored;

[0024] Determine the business type corresponding to the monitoring log;

[0025] Determine an alarm time period and quantity threshold corresponding to the service type;

[0026] Determine the number of new additions to the monitoring log within the alarm time period;

[0027] When the newly added quantity is greater than the quantity threshold, first warning information is output to the user.

[0028] Optionally, the method further includes: outputting second alarm information to the user when there is no new increase in quantity within the alarm time period.

[0029] Optionally, the method further includes:

[0030] Obtaining the log level of the target log and the user authority information of the user;

[0031] Dividing the user authority information into at least one authority information segment; each of the at least one authority information segment corresponds to a different log level query authority;

[0032] Verifying the permission information segment corresponding to the log level of the target log;

[0033] After the permission information fragment corresponding to the log level of the target log is verified, the user is allowed to query the target log.

[0034] On the other hand, an embodiment of the present invention further discloses a log management method and apparatus, the apparatus comprising:

[0035] Keyword acquisition module, used to obtain query keywords for logs;

[0036] A keyword determination module, configured to determine associated keywords based on the query keywords; the associated keywords are generated based on semantic associations of the query keywords;

[0037] A search field determination module, configured to determine an inverted index field and a balanced tree index field from at least one field included in the associative keyword;

[0038] a candidate log determining module, configured to determine at least one candidate log according to the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field;

[0039] a candidate log keyword determination module, configured to respectively determine content keywords of the at least one candidate log;

[0040] a display score determination module, configured to determine a display score for the at least one candidate log based on the content keyword and the association keyword;

[0041] The target log processing module is configured to select, from the at least one candidate log, the log having a display score greater than a preset score threshold as a target log, and output the target log to a user.

[0042] Optionally, the display score determination module includes:

[0043] A matching degree determination submodule, configured to determine a matching degree between the content keyword and the association keyword;

[0044] The score determination submodule is configured to determine a display score of the at least one candidate log according to the content keyword and the matching degree.

[0045] Optionally, the log keyword includes a node identifier, where the node identifier is used to indicate a node that generates the log. The first score determination submodule includes:

[0046] A first frequency acquisition unit is configured to determine a first access frequency of the node within a first preset time period; the first access frequency is used to indicate a frequency at which logs of the node are accessed within the first preset time period;

[0047] The node retrieval score determination unit is configured to determine a display score of the at least one candidate log according to the content keyword, the matching degree, and the first retrieval frequency.

[0048] Optionally, the node call score determination submodule includes:

[0049] A second frequency acquisition unit is used to determine a second retrieval frequency of the log within a second preset time period; the second retrieval frequency is used to indicate the number of times the log is retrieved within the second preset time period;

[0050] A comprehensive score determination unit is configured to determine a display score of the at least one candidate log according to the second retrieval frequency, the matching degree, and the first retrieval frequency.

[0051] Optionally, the device further comprises:

[0052] A monitoring log determination submodule, configured to use the target log as a monitoring log to be monitored after the target log is output to the user;

[0053] A business type determination submodule, used to determine the business type corresponding to the monitoring log;

[0054] An alarm rule determination submodule, configured to determine an alarm time period and quantity threshold corresponding to the service type;

[0055] A new number determination submodule is used to determine the number of new additions to the monitoring log within the alarm time period;

[0056] The first alarm information output submodule is used to output the first alarm information to the user when the newly added number is greater than the number threshold.

[0057] Optionally, the device further comprises:

[0058] The second alarm information output submodule is used to output the second alarm information to the user when there is no new increase in the number within the alarm time period.

[0059] Optionally, the device further comprises:

[0060] The log level and permission acquisition submodule is used to obtain the log level of the target log and the user permission information of the user;

[0061] The authority division submodule is used to divide the user authority information into at least one authority information segment; the at least one authority information segment corresponds to a different log level query authority;

[0062] The permission verification submodule is used to verify the permission information segment corresponding to the log level of the target log;

[0063] The query qualification determination submodule is used to allow the user to query the target log after the permission information segment corresponding to the log level of the target log is verified.

[0064] Accordingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various steps of the above-mentioned log management method embodiment are implemented.

[0065] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned log management method embodiment is implemented.

[0066] Embodiments of the present invention have the following advantages: Associative keywords are generated based on the semantic association of query keywords, enabling the method to better understand the user's query intent and automatically expand related keywords. An inverted index field and a balanced tree index field are determined from at least one field included in the associative keyword, enabling efficient retrieval and location of relevant log entries. Based on the inverted index field and its corresponding inverted index, and the balanced tree index field and its corresponding balanced tree index, at least one candidate log is quickly identified. Subsequently, the content keywords for each candidate log are determined, and a display score for each candidate log is calculated based on the content keywords and associative keywords. Finally, among the candidate logs, logs with a display score greater than a preset score threshold are selected as target logs and output to the user. This not only improves the accuracy and efficiency of log queries but also reduces the tedious manual entry of multiple keywords by users, significantly enhancing the speed and accuracy of fault diagnosis and troubleshooting. Furthermore, by automatically expanding query keywords, the system can more comprehensively cover relevant log entries, reducing the risk of missing important information and further improving the reliability of log management and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of steps of an embodiment of a log management method of the present invention;

[0068] Figure 2 is a flow chart of an embodiment of a log management method of the present invention;

[0069] Figure 3 It is a structural block diagram of an embodiment of a log management method and device of the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] One of the core concepts of this embodiment is to quickly identify at least one candidate log using an inverted index and a balanced tree index. Subsequently, the content keywords of each candidate log are determined, and a display score for each candidate log is calculated based on the content keywords and associated keywords. Finally, among the candidate logs, those with a display score greater than a preset score threshold are selected as target logs and output to the user.

[0072] Reference Figure 1 , shows a flow chart of steps of an embodiment of a log management method of the present invention, which may specifically include the following steps:

[0073] Step 101: Obtain query keywords for the log;

[0074] A log is a record file generated by a system, application, or device during operation, which is used to record events, operations, errors, and other information. The content of a log usually includes key fields such as timestamp, node identifier, and event description. For example, a log format may be: [yyyy-MM-dd hh:mm:ss][Node_001]Error:Payment timeout, where yyyy in yyyy-MM-dd mm:hh:ss represents the year, MM represents the month, dd represents the specific date, hh represents the hour, mm represents the minute, and ss represents the specific second; the time format can be freely set according to business needs, for example, it can be set to a common format such as YYYY-MM-DDTHH:mm:ss.fffZ format, and this application does not limit this.

[0075] Query keywords are key information entered by users to express their search requirements for log content. Query keywords can be specific words, phrases, or conditions, such as "payment failure," "server exception," or "database error." Users can submit queries through the corresponding search device's interface input box, command-line tools, or other interactive methods. The system receives user input and parses it into one or more query keywords.

[0076] Step 102: determining an associative keyword based on the query keyword; the associative keyword is generated based on the semantic association of the query keyword;

[0077] Associated Keywords are related words or phrases generated through semantic analysis based on query keywords. They expand the scope of query keywords and help the system understand user needs more comprehensively.

[0078] For example, the query keyword input by the user is: "database connection timeout", and the associated keywords at this time may include: database, connection pool, timeout error, lock and other keywords.

[0079] This process can be achieved by applying natural language processing technology. In addition to natural language processing technology, it can also be achieved using technologies such as word vector models, industry expertise graphs, context analysis or proprietary dictionaries.

[0080] For example, the query keyword "database connection timeout" can be implemented using the built-in telecom industry-specific dictionary to parse the keyword into a keyword triple of <database, connection pool, timeout>.

[0081] Step 103: determining an inverted index field and a balanced tree index field from at least one field included in the associative keyword;

[0082] Generally, log fields can be divided into the following categories: Text fields: such as message, description, etc., used to store descriptive information. Numeric fields: such as timestamp, duration, etc., used to store numerical data such as timestamp and duration. Identification fields: such as node_id, user_id, etc., used to identify the node or user that generated the log. Inverted Index Fields refers to fields suitable for full-text search. It speeds up the search by mapping keywords in the document to a list of documents containing the keyword. Because text fields usually contain a large amount of unstructured text information, they are suitable for creating inverted indexes. Balanced Tree Index Fields refers to fields suitable for range queries or sorting operations. It supports efficient range scans and single-value searches through ordered data structures such as B-trees and B+ trees. Among them, numeric fields or ordered fields are suitable for creating balanced tree indexes.

[0083] An inverted index is an index structure used for full-text search. It accelerates searches by mapping keywords in a document to a list of documents containing the keyword. Unlike traditional forward indexes, an inverted index maps words to documents in reverse order. A balanced tree index is a data structure based on a balanced tree that supports range queries and sorting operations. By storing data in an ordered manner, it makes range scans and single-value lookups very efficient. For example, if you need to filter logs within a certain time range, the timestamp field might be selected as a balanced tree index field.

[0084] For example, taking the 5GC signaling server log of a data center as an example, one of the processes for creating an inverted index and a balanced index is given:

[0085] First, standardize the format of the original logs, for example, unify the timestamp format in the logs to the target format;

[0086] Extract key information from the standardized log. For example, if a log contains the following key information: timestamp, node IP, event type, error code, and business line ID, you can create an inverted index based on the business line ID and error type. The corresponding index key is the concatenation of the business line ID and error type text, such as concatenation with underscores or other formats. The content of the value corresponding to the key can be associated with the physical storage address of the log.

[0087] Then, a balanced tree index, such as a B-tree index, can be generated based on the timestamp field and divided into time series shards at hourly granularity. The key of the corresponding B-tree index can be a time range identifier, and the value can correspond to the storage location of all logs within the time period.

[0088] Step 104: Determine at least one candidate log according to the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field.

[0089] Intersect the inverted index result set and the balanced tree index result set to obtain a set of logs that meet both the keyword matching and time range conditions. These logs are called candidate logs.

[0090] Step 105 , determining content keywords of the at least one candidate log respectively;

[0091] In some scenarios, the keywords entered by the user may correspond to a wide range of logs. In this case, due to resource constraints and business needs, it is necessary to filter the retrieved candidate logs and select logs that may be more valuable. The key step in determining the value of candidate logs is to determine the content keywords of the candidate logs.

[0092] Step 106: Determine a display score for the at least one candidate log based on the content keyword and the association keyword.

[0093] After obtaining the content keywords of the candidate log, it is necessary to combine them with the associated keywords to determine the score of the candidate log. The score reflects the importance of the candidate log and is also the value of the candidate log.

[0094] In one embodiment, step 106 includes the following sub-steps:

[0095] Sub-step S11, determining the matching degree between the content keyword and the association keyword;

[0096] When considering the value of candidate logs, the first step is to determine whether the candidate logs meet the user's query intent. Therefore, the matching degree between the content keywords and the associated keywords must be confirmed. A high matching degree indicates that the candidate log is likely to meet the user's query intent. This matching degree can be determined by calculating the correlation coefficient between the content keywords and the associated keywords, or by calculating the Euclidean distance or cosine similarity between the content keywords and the associated keywords, which is not limited in this application.

[0097] The specific implementation method of quantitatively judging the matching degree can be to set a matching threshold. When the matching degree value exceeds the set matching threshold, it is considered that the log meets the preliminary requirements, and then subsequent score calculation is performed.

[0098] For example, in the process of calculating the matching degree, the matching degree of certain words may account for a larger proportion than the matching degree of other words. For example, when urgently troubleshooting the root cause of an error, the error code keyword and business line ID may be more important. At this time, when calculating the keyword matching degree, a weight can be set according to business needs, and the matching degree indicators of different keys can be multiplied by the corresponding weight to obtain the final total matching degree; a weight coefficient can also be preset according to the business scenario to naturally amplify the proportion of important keyword types in the business scenario.

[0099] Sub-step S12: determining a display score of the at least one candidate log according to the content keyword and the matching degree.

[0100] In one embodiment, the log keyword includes a node identifier, and the node identifier is used to indicate the node that generates the log. Sub-step S12 may include the following sub-steps:

[0101] Sub-step S121, determining a first retrieval frequency of the node within a first preset time period; the first retrieval frequency is used to represent the frequency at which logs are retrieved by the node within the first preset time period;

[0102] A node ID is information in a log that uniquely identifies the node that generated the log. For example, Node_IP = 10.10.1.23 or Node_ID = Node_001. It helps distinguish logs generated by different nodes, allowing you to monitor and analyze the behavior or status of a specific node.

[0103] The first preset time period is a time window used to collect statistics on node behavior characteristics within a specific time period. The length of the time period can be flexibly set based on business needs. For example, short periods such as 5 minutes or 1 hour are suitable for real-time monitoring scenarios, while long periods such as 1 day or 1 week are suitable for historical data analysis scenarios.

[0104] The first retrieval frequency refers to the number of times a node's logs are retrieved within the first preset time period. The frequency of log retrieval can reflect the importance or degree of anomalies of the node's logs. For example, if a node's logs are frequently retrieved, it may indicate that the node is anomaly or requires special attention. If a node's logs are rarely retrieved, it may indicate that the node is relatively stable.

[0105] In log analysis systems, the node ID is a crucial field that helps the system quickly locate the source of logs. The primary call frequency of a statistical node is used to assess the node's activity or anomaly within a specific time period.

[0106] Sub-step S122: Determine the display score of the at least one candidate log based on the content keyword, the matching degree, and the first access frequency. After obtaining the first access frequency, subsequent calculations can be performed based on the frequency.

[0107] In one embodiment, sub-step S122 may include the following sub-steps:

[0108] Sub-step S1221, determining a second retrieval frequency of the log within a second preset time period; the second retrieval frequency is used to indicate the number of times the log is retrieved within the second preset time period;

[0109] The second preset time period is a time window used to count the number of times a log entry is retrieved within a specific time period. The specific setting method can be the same as the first preset time period above, and will not be repeated here.

[0110] The second access frequency refers to the number of times a log entry is accessed by a user or the system within a second preset time period. It reflects the frequency of use or attention paid to the log entry within a specific time period. For example, if a log entry is frequently accessed, it may indicate that its content is highly valuable or relevant to the user. If a log entry is rarely accessed, it may indicate that its content is relatively minor or unimportant.

[0111] Each log entry has a unique identifier that distinguishes different log records. When calculating the second access frequency, accurate matching is required based on the log's unique identifier. In large-scale log systems, the second access frequency can be used to identify hot logs. For example, if a log entry has a significantly higher second access frequency than other logs, this may indicate that the log is currently hot content. Alerts or automated analysis can be triggered based on hot logs.

[0112] Sub-step S1222 : determining a display score of the at least one candidate log according to the second retrieval frequency, the matching degree, and the first retrieval frequency.

[0113] For example, the display score may be calculated according to the second access frequency, the matching degree, and the first access frequency using the following formula:

[0114] S=ω N ·N f +ω L ·L f +ω K ·K s , where S represents the display score, K s Represents the matching degree, ω K Represents the preset weight corresponding to the matching degree, L frepresents the second tuning frequency, ω L Represents the preset weight corresponding to the second access frequency, N f represents the first tuning frequency, ω N Represents the preset weight corresponding to the first access frequency.

[0115] The corresponding weight settings can be flexibly set according to business needs, and this application does not limit this.

[0116] Step 107 : Among the at least one candidate log, the log having a presentation score greater than a preset score threshold is selected as a target log, and the target log is output to the user.

[0117] In addition, when outputting logs, you can filter and display them by log level, allowing users to selectively view log entries of different levels based on business needs. The introduction of log highlighting and color coding helps users more quickly identify different types of log entries.

[0118] In one embodiment, the method may further include the following:

[0119] After outputting the target log to the user, using the target log as a monitoring log to be monitored;

[0120] Determine the business type corresponding to the monitoring log;

[0121] Determine an alarm time period and quantity threshold corresponding to the service type;

[0122] Determine the number of new additions to the monitoring log within the alarm time period;

[0123] When the newly added quantity is greater than the quantity threshold, first warning information is output to the user.

[0124] In one embodiment, the method may further include the following:

[0125] In the case that there is no new increase in quantity within the alarm time period, second alarm information is output to the user.

[0126] In a log management system, users often need to monitor specific logs and determine whether system anomalies are occurring based on changes in these logs. The purpose of the alert mechanism is to quickly identify potential issues and alert users by monitoring log changes in real time. After being output to the user, the target log is marked as requiring monitoring. Alert intervals and quantity thresholds are set based on the business type of the monitoring log. New entries are counted within the alert interval, and different alert messages are triggered based on the relationship between the new entry count and the threshold.

[0127] If the number of new monitoring logs exceeds a preset threshold, a first alarm message is triggered. The first alarm message is usually used to remind the user that there may be sudden abnormalities or high-risk events in the system.

[0128] If the monitoring log does not increase in the alarm time period, the second alarm message is triggered. The second alarm message is usually used to remind the user that the system may have silent abnormalities (such as node downtime, log loss, etc.).

[0129] For example, suppose you are monitoring the 5GC signaling server logs of a distributed system. A user queries logs containing the error code 0xE12A and marks these logs as target logs. The system identifies these target logs as monitoring logs and determines the corresponding service type based on keywords in the log content. Based on the service type, corresponding alarm rules are set. For example, if the same error code appears ≥50 times within 5 minutes, a Level 1 alarm is triggered.

[0130] If there is no new log for error code 0xE12A within a certain alarm time period, the second alarm message will be triggered. The following alarm information can be pushed to the client interface in real time via WebSocket:

[0131] For example, a complete alarm example process can be as follows:

[0132] First, set up database-level alert rules. For example, based on the preset rule "the same error code appears ≥50 times within 5 minutes," a Level 1 alert is triggered. Furthermore, statistical models can be used for early warning, such as using an ARIMA time series model to predict log volume fluctuations. For example, the predicted log volume for 10:00 AM to 11:00 AM might be 50,000, but the actual volume reached 160,000, exceeding the predicted value by three standard deviations. This could be flagged as a potential resource bottleneck and an alert generated. Finally, deep learning methods can be used for in-depth alert processing, such as analyzing log interval patterns using an LSTM neural network.

[0133] In one embodiment, the method may further include the following:

[0134] Obtaining the log level of the target log and the user authority information of the user;

[0135] Dividing the user authority information into at least one authority information segment; each of the at least one authority information segment corresponds to a different log level query authority;

[0136] Verifying the permission information segment corresponding to the log level of the target log;

[0137] After the permission information fragment corresponding to the log level of the target log is verified, the user is allowed to query the target log.

[0138] In a log management system, different users may have different permission levels, which determine which logs they can access. To ensure data security and accurate permission control, the system needs to fine-grainedly classify user permission information and verify whether the user has access rights based on the log level of the target log.

[0139] Log levels are used to identify the importance or sensitivity of logs, such as INFO, WARN, and ERROR. User permission information can include the user's role and permission scope. User permission information is segmented into multiple segments, each corresponding to a specific log level query permission. For example, permission information can be segmented into L1-L5, corresponding to different approval nodes or permission scopes. Based on the log level of the target log, the corresponding permission information segment is located. The segment is then verified for legitimacy. If verification passes, the user is allowed to query the target log; otherwise, access is denied.

[0140] For example, a user accesses the log management system through a browser, and the browser sends an HTTP request to the server, requesting a list of applications in the cluster. The browser uses encryption technology based on permission sharding to protect the request: the user permission information is as follows: user role: operation and maintenance engineer; permission scope: Zhejiang Telecom Core Network. The user permission information is converted into a 32-bit binary code and divided into L1-L5 segments according to the preset 5 approval nodes: each segment is attached with a unique identification code, such as 1L1 corresponds to the approval number of node a1, and an encrypted string is generated: the encrypted string is encrypted using the AES-256 algorithm and transmitted as the request header x-auth-token. After the server receives the request, it first reverse-parses the encrypted string: the system locates the corresponding permission information segment based on the log level and module type of the target log. If the verification passes, the user is allowed to query the target log; otherwise, access is denied.

[0141] Furthermore, the method described in the present invention can also adopt a client-server architecture, in which the client is responsible for sending requests, receiving responses, and displaying log content, while the server is responsible for processing requests, retrieving log data, and returning results. To improve request processing efficiency, paging mechanisms and fuzzy search functions can also be introduced. When sending a request, the client can specify a page number and search keywords. The server uses this information to retrieve the corresponding log data and return the results. To reduce the number of server requests and increase data loading speed, an intelligent caching strategy can also be adopted. After the client initially obtains log data, it caches it in a local or remote cache service and prioritizes retrieval from the cache in subsequent requests. Furthermore, incremental updates are implemented based on identifiers such as timestamps or log IDs, ensuring that users can view the latest log content. Encryption technology based on the requester's permission information can also be used to protect transmitted log data, ensuring data confidentiality and integrity. The log viewing function can also be designed as a pluggable module, allowing for easy integration into different systems. A rich set of interfaces and configuration options are also provided for administrators to adjust and optimize operations to meet the needs of different scenarios. In addition, a modular design concept is adopted to divide the system into multiple independent functional modules for development and implementation to reduce system complexity and improve maintainability.

[0142] Associative keywords are generated based on the semantic associations of query keywords, enabling this method to better understand the user's query intent and automatically expand relevant keywords. An inverted index field and a balanced tree index field are determined from at least one field included in the associative keyword, enabling efficient retrieval and location of relevant log entries. Based on the inverted index field and its corresponding inverted index, and the balanced tree index field and its corresponding balanced tree index, at least one candidate log is quickly identified. Subsequently, the content keywords for each candidate log are determined, and a display score for each candidate log is calculated based on the content keywords and associative keywords. Finally, among the candidate logs, logs with a display score greater than a preset threshold are selected as target logs and output to the user. This not only improves the accuracy and efficiency of log queries but also reduces the tedious manual entry of multiple keywords by users, significantly enhancing the speed and accuracy of fault diagnosis and troubleshooting. Furthermore, by automatically expanding query keywords, the system can more comprehensively cover relevant log entries, reducing the risk of missing important information and further improving the reliability of log management and user experience.

[0143] Reference Figure 2 , shows a flow chart of an embodiment of a calendar management method of the present invention. The log management method of the present invention can be used on a server cluster. The specific process can be:

[0144] First, obtain the logs of each node in the cluster, standardize the format of the obtained logs according to the above content, and integrate them together. One way of integration can be to use the LZ77-Huffman hybrid compression algorithm to encode and compress repeated log content such as periodic heartbeat logs, and use the SimHash algorithm to deduplicate logs with similarity exceeding the threshold; after integration, formulate the fields for establishing inverted index and balanced tree index according to business needs, and create indexes; after the user initiates a query request, first obtain preliminary results through the user's query keywords and preset indexes, and then filter the logs in the preliminary results. The detailed process has been described above and will not be repeated here. After completing the screening of the preliminary results, the screened logs will be output to the user. Before outputting them to the user, the user's permissions can also be verified in combination with the above-mentioned permission verification content.

[0145] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0146] Reference Figure 3 , shows a structural block diagram of a log management method and device embodiment of the present invention, which may specifically include the following modules:

[0147] Keyword acquisition module 201, used to obtain query keywords for logs;

[0148] The keyword determination module 202 is configured to determine associated keywords based on the query keywords; the associated keywords are generated based on the semantic association of the query keywords;

[0149] A search field determination module 203 is configured to determine an inverted index field and a balanced tree index field from at least one field included in the associative keyword;

[0150] a candidate log determining module 204, configured to determine at least one candidate log based on the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field;

[0151] a candidate log keyword determination module 205 , configured to respectively determine content keywords of the at least one candidate log;

[0152] a display score determination module 206, configured to determine a display score for the at least one candidate log based on the content keyword and the association keyword;

[0153] The target log processing module 207 is configured to select, from the at least one candidate log, the log having a display score greater than a preset score threshold as a target log, and output the target log to the user.

[0154] In one embodiment, the display score determination module includes:

[0155] A matching degree determination submodule, configured to determine a matching degree between the content keyword and the association keyword;

[0156] The score determination submodule is configured to determine a display score of the at least one candidate log according to the content keyword and the matching degree.

[0157] In one embodiment, the log keyword includes a node identifier, where the node identifier is used to indicate the node that generated the log. The first score determination submodule includes:

[0158] A first frequency acquisition unit is configured to determine a first access frequency of the node within a first preset time period; the first access frequency is used to indicate a frequency at which logs of the node are accessed within the first preset time period;

[0159] The node retrieval score determination unit is configured to determine a display score of the at least one candidate log according to the content keyword, the matching degree, and the first retrieval frequency.

[0160] In one embodiment, the node calls the score determination submodule, including:

[0161] A second frequency acquisition unit is used to determine a second retrieval frequency of the log within a second preset time period; the second retrieval frequency is used to indicate the number of times the log is retrieved within the second preset time period;

[0162] A comprehensive score determination unit is configured to determine a display score of the at least one candidate log according to the second retrieval frequency, the matching degree, and the first retrieval frequency.

[0163] In one embodiment, the apparatus further comprises:

[0164] A monitoring log determination submodule, configured to use the target log as a monitoring log to be monitored after the target log is output to the user;

[0165] A business type determination submodule, used to determine the business type corresponding to the monitoring log;

[0166] An alarm rule determination submodule, configured to determine an alarm time period and quantity threshold corresponding to the service type;

[0167] A new number determination submodule is used to determine the number of new additions to the monitoring log within the alarm time period;

[0168] The first alarm information output submodule is used to output the first alarm information to the user when the newly added number is greater than the number threshold.

[0169] In one embodiment, the apparatus further comprises:

[0170] The second alarm information output submodule is used to output the second alarm information to the user when there is no new increase in the number within the alarm time period.

[0171] In one embodiment, the apparatus further comprises:

[0172] The log level and permission acquisition submodule is used to obtain the log level of the target log and the user permission information of the user;

[0173] The authority division submodule is used to divide the user authority information into at least one authority information segment; the at least one authority information segment corresponds to a different log level query authority;

[0174] The permission verification submodule is used to verify the permission information segment corresponding to the log level of the target log;

[0175] The query qualification determination submodule is used to allow the user to query the target log after the permission information segment corresponding to the log level of the target log is verified.

[0176] Associative keywords are generated based on the semantic associations of query keywords, enabling this method to better understand the user's query intent and automatically expand relevant keywords. An inverted index field and a balanced tree index field are determined from at least one field included in the associative keyword, enabling efficient retrieval and location of relevant log entries. Based on the inverted index field and its corresponding inverted index, and the balanced tree index field and its corresponding balanced tree index, at least one candidate log is quickly identified. Subsequently, the content keywords for each candidate log are determined, and a display score for each candidate log is calculated based on the content keywords and associative keywords. Finally, among the candidate logs, logs with a display score greater than a preset threshold are selected as target logs and output to the user. This not only improves the accuracy and efficiency of log queries but also reduces the tedious manual entry of multiple keywords by users, significantly enhancing the speed and accuracy of fault diagnosis and troubleshooting. Furthermore, by automatically expanding query keywords, the system can more comprehensively cover relevant log entries, reducing the risk of missing important information and further improving the reliability of log management and user experience.

[0177] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0178] An embodiment of the present invention also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned log management method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0179] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned log management method embodiment are implemented and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0181] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0185] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0186] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0187] The above is a detailed introduction to the log management method, device, equipment and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A log management method, characterized in that: The method comprises: Get the query keywords for the log; Determining an associative keyword based on the query keyword; the associative keyword is generated based on the semantic association of the query keyword; Determine an inverted index field and a balanced tree index field from at least one field included in the associative keyword; Determine at least one candidate log according to the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field; respectively determining content keywords of the at least one candidate log; determining a display score of the at least one candidate log according to the content keyword and the association keyword; Among the at least one candidate log, the log having a presentation score greater than a preset score threshold is selected as a target log, and the target log is output to the user.

2. The log management method according to claim 1, wherein: Determining the display score of the at least one candidate log based on the content keyword and the association keyword includes: Determining a matching degree between the content keyword and the associated keyword; A display score of the at least one candidate log is determined according to the content keyword and the matching degree.

3. The log management method according to claim 2, characterized in that: The log keyword includes a node identifier, where the node identifier is used to indicate a node that generates the log. The determining of a display score of the at least one candidate log based on the content keyword and the matching degree includes: Determine a first retrieval frequency of the node within a first preset time period; the first retrieval frequency is used to indicate the frequency at which logs of the node are retrieved within the first preset time period; A display score of the at least one candidate log is determined according to the content keyword, the matching degree, and the first retrieval frequency.

4. The log management method according to claim 3, wherein: Determining a display score of the at least one candidate log according to the content keyword, the matching degree, and the first access frequency includes: Determining a second access frequency of the log within a second preset time period; the second access frequency is used to indicate the number of times the log is accessed within the second preset time period; A display score of the at least one candidate log is determined according to the second retrieval frequency, the matching degree, and the first retrieval frequency.

5. The log management method according to claim 1, wherein: The method further comprises: After outputting the target log to the user, using the target log as a monitoring log to be monitored; Determine the business type corresponding to the monitoring log; Determine an alarm time period and quantity threshold corresponding to the service type; Determine the number of new additions to the monitoring log within the alarm time period; When the newly added quantity is greater than the quantity threshold, first warning information is output to the user.

6. The log management method according to claim 5, characterized in that: The method further includes: outputting second warning information to the user when there is no new increase in the warning time period.

7. The log management method according to claim 1, wherein: The method further comprises: Obtaining the log level of the target log and the user authority information of the user; Dividing the user authority information into at least one authority information segment; each of the at least one authority information segment corresponds to a different log level query authority; Verifying the permission information segment corresponding to the log level of the target log; After the permission information fragment corresponding to the log level of the target log is verified, the user is allowed to query the target log.

8. A log management method and device, characterized in that: The device comprises: Keyword acquisition module, used to obtain query keywords for logs; A keyword determination module, configured to determine associated keywords based on the query keywords; the associated keywords are generated based on semantic associations of the query keywords; A search field determination module, configured to determine an inverted index field and a balanced tree index field from at least one field included in the associative keyword; a candidate log determining module, configured to determine at least one candidate log according to the inverted index field, the inverted index corresponding to the inverted index field, the balanced tree index field, and the balanced tree index corresponding to the balanced tree index field; a candidate log keyword determination module, configured to respectively determine content keywords of the at least one candidate log; a display score determination module, configured to determine a display score for the at least one candidate log based on the content keyword and the association keyword; The target log processing module is configured to select, from the at least one candidate log, the log having a display score greater than a preset score threshold as a target log, and output the target log to a user.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the log management method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log management method according to any one of claims 1 to 7 are implemented.

Images