Log extraction method and device based on service attribute, equipment and storage medium

Abstract

The application belongs to the technical field of computers, and particularly relates to a log extraction method and device based on business attributes, equipment and a storage medium. The method comprises the following steps: obtaining an attribute range of a log to be extracted; obtaining a storage location of the log to be extracted according to the attribute range; generating a log extraction request corresponding to the storage location; the log extraction request is used to indicate a storage level of the log to be extracted; different storage locations store logs corresponding to different attribute ranges; and the log to be extracted is extracted from the storage location according to the storage level. The method improves the efficiency of log query.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for log extraction based on business attributes. Background Technology

[0002] A log file is a file that records events and activities that occur during the runtime of a system or application. System errors, warnings, messages, and debugging information are all stored in log files, making them crucial for diagnosing and resolving system problems. Log files are typically stored on a local disk or a remote server.

[0003] In existing technologies, log files are typically classified according to log directories, indexed, and then stored as files. When a user queries a log file, the search is performed in the stored log file library and the results are sorted.

[0004] However, the existing technologies offer relatively limited log classification and storage methods, resulting in low search efficiency and accuracy when users query logs; the limited file storage format also puts significant pressure on storage space. Summary of the Invention

[0005] This application provides a log extraction method, apparatus, device, and storage medium based on business attributes, to solve the problems of the relatively simple log classification and storage methods provided by the prior art, the low search efficiency and low query accuracy when users query logs, and the problem that the single file saving format will cause great pressure on storage space.

[0006] Firstly, this application provides a log extraction method based on business attributes, including:

[0007] Get the attribute range of the log to be extracted;

[0008] The storage location of the log to be extracted is obtained according to the attribute range, and a log extraction request corresponding to the storage location is generated. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges.

[0009] The log to be retrieved is extracted from the storage location according to the storage level.

[0010] Optionally, the attribute range includes: the query period, the business attribute, and the business attribute value corresponding to the business attribute. Obtaining the storage location of the log to be extracted based on the attribute range includes:

[0011] Obtain the first mapping relationship stored in the database, which is used to indicate the log index and the storage location of the log in the object storage server;

[0012] The log index of the log to be extracted is determined based on the query period, business attributes, and the business attribute values ​​corresponding to the business attributes.

[0013] Based on the log index of the log to be extracted and the first mapping relationship, the storage location of the log to be extracted in the object storage server is determined.

[0014] Optionally, retrieving the log to be retrieved from the storage location according to the storage level includes:

[0015] Based on the storage level, the log to be extracted is extracted from the target area in the storage location, where the target area is the area in the storage location used to store the logs corresponding to the storage level.

[0016] Optionally, before obtaining the attribute range of the log to be extracted, the method further includes:

[0017] Log files are periodically retrieved from the log collector and parsed to obtain the attribute range and storage level of the log files.

[0018] Based on the attribute range corresponding to the log file, generate the log index of the log file and determine the storage location corresponding to the attribute range;

[0019] Based on the storage level corresponding to the log file and the log index, the log files are classified and stored in the corresponding storage locations in the object storage server, and the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log file is stored in the database.

[0020] Optionally, after retrieving the log to be retrieved from the storage location according to the storage level, the method further includes:

[0021] Obtain the exception logs from the logs to be extracted;

[0022] The exception logs are indexed, and the second mapping relationship between the storage location of the exception logs and the log index of the exception logs is stored in the database;

[0023] Based on the anomaly log, perform event analysis and processing to obtain the analysis results.

[0024] Optionally, after performing event analysis processing based on the exception log and obtaining the analysis results, the method further includes:

[0025] Determine whether the analysis results can identify the cause of the anomaly.

[0026] If so, output the aforementioned exception log;

[0027] If not, reset the attribute range of the log to be extracted, and extract the new exception log from the storage location corresponding to the new attribute range;

[0028] Based on the new anomaly log, event analysis processing is performed to obtain new analysis results until the new analysis results can determine the cause of the anomaly.

[0029] Secondly, this application provides a log extraction device based on business attributes, comprising:

[0030] The acquisition module is used to obtain the attribute range of the log to be extracted;

[0031] The generation module is used to obtain the storage location of the log to be extracted according to the attribute range, and generate a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges.

[0032] The processing module is used to extract the log to be extracted from the storage location according to the storage level.

[0033] Optionally, the log extraction device based on business attributes further includes: a determination module;

[0034] The acquisition module is further configured to acquire a first mapping relationship stored in the database, the first mapping relationship being used to indicate the log index and the storage location of the log in the object storage server;

[0035] The determining module is used to determine the log index of the log to be extracted based on the query time period, business attributes, and the business attribute values ​​corresponding to the business attributes.

[0036] The determining module is further configured to determine the storage location of the log to be extracted in the object storage server based on the log index of the log to be extracted and the first mapping relationship.

[0037] Optionally, the processing module is specifically used to extract the log to be extracted from a target area in the storage location according to the storage level, wherein the target area is the area in the storage location used to store the logs corresponding to the storage level.

[0038] Optionally, the processing module is also used to periodically retrieve log files from the log collector, parse the log files, and obtain the attribute range and storage level corresponding to the log files.

[0039] The generation module is further configured to generate a log index for the log file based on the attribute range corresponding to the log file, and determine the storage location corresponding to the attribute range;

[0040] The processing module is further configured to classify and store the log files to corresponding storage locations in the object storage server according to the storage level corresponding to the log files and the log index, and store the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log files in the database.

[0041] Optionally, the acquisition module is further configured to acquire abnormal logs from the logs to be extracted;

[0042] The processing module is also used to index the exception log and store the second mapping relationship between the storage location of the exception log and the log index of the exception log in the database;

[0043] The processing module is also used to perform event analysis processing based on the exception log to obtain analysis results.

[0044] Optionally, the log extraction device based on business attributes further includes: a judgment module;

[0045] The judgment module is used to determine whether the analysis results can identify the cause of the anomaly.

[0046] The processing module is also used to output the anomaly log when the analysis results can determine the cause of the anomaly.

[0047] The processing module is also used to reset the attribute range of the log to be extracted and extract new abnormal logs from the storage location corresponding to the new attribute range when the analysis results cannot determine the cause of the abnormality.

[0048] The processing module is also used to perform event analysis processing based on the new anomaly log to obtain new analysis results until the new analysis results can determine the cause of the anomaly.

[0049] Thirdly, this application provides a log extraction device based on business attributes, comprising:

[0050] Memory;

[0051] processor;

[0052] The memory stores computer-executed instructions;

[0053] The processor executes computer execution instructions stored in the memory to implement the log extraction method based on business attributes as described in the first aspect and various possible implementations of the first aspect above.

[0054] Fourthly, this application provides a computer storage medium, characterized in that the computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the log extraction method based on business attributes as described in the first aspect and various possible implementations of the first aspect.

[0055] The log extraction method based on business attributes provided in this application obtains the attribute range of the log to be extracted, obtains the storage location of the log to be extracted according to the attribute range, and generates a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges. According to the storage level, the log to be extracted is extracted from the storage location. This method improves the efficiency of log querying. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 The process of the log extraction method based on business attributes provided in this application Figure 1 ;

[0058] Figure 2 The process of the log extraction method based on business attributes provided in this application Figure 2

[0059] Figure 3 A schematic diagram of the log extraction device based on business attributes provided in this application;

[0060] Figure 4 A schematic diagram of the log extraction device based on business attributes provided in this application.

[0061] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0064] In real-world scenarios, log files are software-generated files containing information about the operation, activities, and usage patterns of applications, servers, or IT systems. Log files contain a historical record of all processes, events, and messages, along with other descriptive data to provide context for this information. Timestamps show what happened internally within the system and when it occurred. When abnormal events occur in the system, log files can be used to locate and analyze the anomalies.

[0065] However, because log files are constantly being generated and the data volume is enormous, when a problem occurs and the corresponding file needs to be found, only a simple search can be performed according to the log directory. The search results are only sorted, greatly increasing the difficulty of querying log files. Therefore, to quickly locate the log file for an anomaly, the log file search logic needs to be improved. How to improve the efficiency of log file searches is the problem that needs to be solved.

[0066] To address the aforementioned issues, this application proposes a log extraction method based on business attributes. This method assigns storage levels to log files and categorizes them according to business attributes, establishing a mapping between log indexes and log storage locations to quickly locate log files, thereby improving the speed of searching for log files.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] Figure 1 The process of the log extraction method based on business attributes provided in this application Figure 1 .like Figure 1 As shown in this embodiment, the log extraction method based on business attributes includes:

[0069] S101: Get the attribute range of the log to be extracted.

[0070] The attribute range indicates the characteristics of the log to be extracted, which correspond to the characteristics set when the log was stored. The attribute range can include three parts: the query period, the business attribute, and the business attribute value. The log to be extracted will contain relevant information such as the period, business attribute, and business attribute value.

[0071] Understandably, there are two reasons for obtaining the attribute range of the logs to be extracted: one is to determine the user's query requirements, and the other is to quickly find the log content that the user needs based on the main characteristics of the logs, that is, to enable the user to efficiently search for the log content they need based on the attribute range.

[0072] For example, the attribute range includes: query period, business attribute, and business attribute value. The query period can be: 13:00-14:00 on September 25, 2023; the business attribute can be: campus card; and the business attribute value can be: number segment 155. When a user enters: query period: 13:00-14:00 on September 25, 2023, business attribute: campus card, and business attribute value: number segment 155, logs with the corresponding characteristics can be retrieved.

[0073] S102: Obtain the storage location of the log to be extracted according to the attribute range, and generate a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges.

[0074] The storage location can be an object storage server or a database. A log retrieval request indicates to a querying user that they need to retrieve logs; the user wants to search for or extract the desired log file. Storage level is a storage tier set by the log indexer based on the severity of log anomalies; storage levels are used to determine where the logs will be stored.

[0075] Understandably, based on the input attribute range, the index information recorded in the database can be queried, thereby enabling the extraction of the log file's location in object storage. In other words, logs generated by different applications are uploaded to a log collector, which categorizes the log files corresponding to different applications. The log indexer then extracts log features from these categorized log files, builds and generates index data, and stores it in the database. When a user needs to extract or query logs from the database, they can input a range of log attributes. The index will then retrieve the corresponding log storage location and generate a log extraction request for that location.

[0076] The log indexer extracts log features from different categories of log files and stores them according to dimensions, ultimately storing all logs in object storage or a database. For example, dimensions could include: province-based logs, user-based logs, business process logs, API / front-end page logs, business exception logs, and system exception logs. Different log indexers extract log features and store the logs based on those features.

[0077] S103: Extract the log to be extracted from the storage location according to the storage level.

[0078] The log retrieval request includes the storage level. The storage level of the logs to be retrieved is not set by the user, but determined by the indexer based on the log's attribute range. For example, the storage level can be either object storage or database. Object storage will contain province-level logs, user-level logs, business process logs, and interface / front-end page logs; the database will contain business exception logs and system exception logs.

[0079] For example, storage levels can be: Object Storage - Province-level logs; Object Storage - User-level logs; Object Storage - Business Process logs; Object Storage - Interface / Page Frontend logs; Database - Business Exception Logs; Database - System Exception Logs; Database.

[0080] The log extraction method based on business attributes provided in this embodiment obtains the attribute range of the log to be extracted, obtains the storage location of the log to be extracted according to the attribute range, and generates a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges. According to the storage level, the log to be extracted is extracted from the storage location. This method improves the efficiency of log querying.

[0081] Figure 2 The flow of the log extraction method based on business attributes provided in the embodiments of this application Figure 2 .like Figure 2 As shown, this embodiment is... Figure 1 Based on the embodiments, the log extraction method based on business attributes is described in detail. The log extraction method based on business attributes shown in this embodiment includes:

[0082] S201: Periodically retrieve log files from the log collector, parse the log files, and obtain the attribute range and storage level corresponding to the log files.

[0083] The first step in log management is collecting log data. Log collection can be a challenging task because some systems (such as firewalls, intrusion detection systems, and intrusion prevention systems) generate large amounts of log data in EPS (events per second). Organizations need robust log collection mechanisms to collect and process log data, regardless of the volume of log data or the number of devices in the network.

[0084] Each network has different systems and environments that generate various log formats, such as event logs, system logs, and other application logs. Information obtained from router logs differs from information obtained from firewalls. Furthermore, some logs cannot be collected directly, such as logs in the DMZ. In short, log collectors need to be flexible enough to adapt to all network devices and applications.

[0085] Understandably, since log generation takes time, real-time log retrieval may result in multiple duplicate or incomplete log entries. This can lead to a large number of incomplete logs in the log collector, causing multiple logs to appear during subsequent retrieval, only one of which is complete. Users need to examine each log entry individually to determine the correct one to search for. Therefore, log files can be periodically retrieved from the log collector, and then parsed to obtain the attribute range and storage level of the log file.

[0086] S202: Generate a log index for the log file based on the attribute range corresponding to the log file, and determine the storage location corresponding to the attribute range.

[0087] Each log file has its own attribute range, which can be the same or different for each log file. To facilitate user querying of log files, the number of log files that can be searched for for each attribute range is relatively small.

[0088] When the attribute range corresponding to the log file is known (input time period, business attribute, business attribute value), a log index for the corresponding log file will be generated. In other words, the input time period, business attribute, and business attribute value can be the log index of the log file. The storage location of the log file is determined based on the log index, which makes it easier to retrieve the log file.

[0089] S203: Based on the storage level corresponding to the log file and the log index, classify and store the log file to the corresponding storage location in the object storage server, and store the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log file in the database.

[0090] The first mapping relationship is the mapping between the storage location of logs in object storage and the log index. The log index is an index of the log file according to the defined business attribute dimensions. The log index of the log to be extracted is determined by the query period, business attributes, and the corresponding business attribute values.

[0091] Understandably, by parsing the storage level and log index of the log file, the log file is stored in the corresponding storage location on the object storage server according to the dimensions. The association between the corresponding storage location and the log index is then established, and the established data is stored in the database.

[0092] S204: Get the attribute range of the log to be extracted.

[0093] Step S204 is similar to step S101, and will not be described again here.

[0094] S205: Retrieve the first mapping relationship stored in the database.

[0095] The first mapping relationship has already determined the association between the storage location and the log index when the log collector collects logs.

[0096] For example: if the storage location is object storage - province dimension log and log index is from 13:00 on September 25, 2023 to 14:00 on September 25, 2023; there is a relationship between campus card and number segment 155, then this relationship will be in the first mapping relationship.

[0097] S206: Determine the log index of the log to be extracted based on the query period, business attribute, and the business attribute value corresponding to the business attribute.

[0098] For example: the query period can be: 13:00-14:00 on September 25, 2023; the business attribute can be: campus card; the business attribute value can be: number segment 155. The log index for the logs to be extracted is: 13:00-14:00 on September 25, 2023; campus card; number segment 155.

[0099] S207: Determine the storage location of the log to be extracted in the object storage server based on the log index of the log to be extracted and the first mapping relationship.

[0100] Given the log index and the first mapping relationship, the storage location of the log to be extracted can be determined.

[0101] For example: If the log index is from 13:00 to 14:00 on September 25, 2023; campus card; number segment 155; the first mapping relationship is: storage location is object storage - province dimension log. If there is an association between the log index from 13:00 to 14:00 on September 25, 2023; campus card; number segment 155, then this association relationship will be in the first mapping relationship; then the storage location of the log to be extracted in the object storage server is object storage - province dimension log.

[0102] Optionally, the logs to be extracted are extracted from a target area in the storage location according to the storage level, wherein the target area is the area in the storage location used to store logs corresponding to the storage level.

[0103] S208: Obtain the exception logs from the logs to be extracted.

[0104] The exception log is used to indicate logs that are not functioning correctly. An exception log does not mean the log is of low importance; rather, it means that the log is not functioning correctly at this time.

[0105] S209: Index the exception log and store the second mapping relationship between the storage location of the exception log and the log index of the exception log in the database.

[0106] This involves indexing the exception logs and storing the association between the exception log index and its storage location in the value database.

[0107] S210: Based on the aforementioned exception log, perform event analysis and processing to obtain the analysis results.

[0108] There are two main methods for preprocessing unstructured logs. The first, and most common, method uses a parser to extract a unique event identifier and event parameter values ​​from each log entry. Extraction methods include using existing patterns from a known dataset and generating patterns using parsing algorithms such as Drain or Spell. The second method is token-based, splitting the log into a list of words, such as by spaces, and then removing special characters, such as numbers.

[0109] This step involves performing event analysis processing on the exception logs, thereby analyzing the causes of the exceptions in the logs and obtaining the analysis results.

[0110] S211: Determine whether the analysis results can identify the cause of the anomaly; if yes, proceed to step S212; otherwise, proceed to step S213.

[0111] In this process, by determining the cause of the anomaly in the anomaly log, the actual attribute range corresponding to the anomaly log is determined.

[0112] Understandably, when the analysis results can determine the cause of the anomaly, the anomaly log is output; when the analysis results cannot determine the cause of the anomaly, the attribute range of the log to be extracted is reset, and a new anomaly log is extracted from the storage location corresponding to the new attribute range.

[0113] S212: Output the aforementioned exception log.

[0114] When the analysis results can determine the cause of the anomaly, the anomaly log is output. Once the cause of the anomaly is determined, the anomaly log can be output based on the input attribute range.

[0115] S213: Reset the attribute range of the log to be extracted, and extract the new exception log from the storage location corresponding to the new attribute range.

[0116] When the analysis results cannot determine the cause of the anomaly, the attribute range of the log to be extracted is reset, and a new anomaly log is extracted from the storage location corresponding to the new attribute range. When the cause of the anomaly is determined, if the corresponding anomaly log cannot be output based on the input attribute range, the attribute range corresponding to the anomaly log needs to be reset so that the anomaly log can be output after the user inputs the attribute range.

[0117] S214: Based on the new anomaly log, perform event analysis processing to obtain new analysis results until the new analysis results can determine the cause of the anomaly.

[0118] This step involves performing event analysis on the new exception logs, and then analyzing the cause of the exception in the new exception logs to obtain the reason for the existence of the exception logs, so that users can query all logs in the database by attribute range.

[0119] The log extraction method based on business attributes provided in this embodiment periodically retrieves log files from a log collector, parses the log files to obtain the attribute range and storage level corresponding to the log files, generates a log index for the log files based on the attribute range, determines the storage location corresponding to the attribute range, and categorizes and stores the log files to corresponding storage locations in an object storage server based on the storage level and the log index. A first mapping relationship between the storage location corresponding to the attribute range and the log index of the log files is stored in a database. The method then obtains the attribute range of the log to be extracted, retrieves the first mapping relationship stored in the database, and determines the logs to be extracted based on the query period, business attributes, and the corresponding business attribute values. The method involves using a log index to determine the storage location of the log to be extracted in the object storage server based on the log index and the first mapping relationship. It then retrieves abnormal logs from the logs to be extracted, indexes these abnormal logs, and stores the second mapping relationship between the storage location of the abnormal logs and their log index in the database. Based on the abnormal logs, event analysis is performed to obtain analysis results. The method determines whether the analysis results can identify the cause of the abnormality, outputs the abnormal logs, resets the attribute range of the logs to be extracted, and extracts new abnormal logs from the storage location corresponding to the new attribute range. Based on the new abnormal logs, event analysis is performed to obtain new analysis results, until the new analysis results can identify the cause of the abnormality. This method improves the efficiency of log querying.

[0120] Figure 3 A schematic diagram of the log extraction device based on business attributes provided in this application. Figure 3 As shown, the log extraction device 300 based on business attributes provided in this embodiment includes:

[0121] The acquisition module 301 is used to acquire the attribute range of the log to be extracted;

[0122] The generation module 302 is used to obtain the storage location of the log to be extracted according to the attribute range, and generate a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges.

[0123] Processing module 303 is used to extract the log to be extracted from the storage location according to the storage level.

[0124] Optionally, the log extraction device based on business attributes further includes: a determination module 304;

[0125] The acquisition module 301 is further configured to acquire a first mapping relationship stored in the database, the first mapping relationship being used to indicate the log index and the storage location of the log in the object storage server;

[0126] The determining module 304 is used to determine the log index of the log to be extracted based on the query period, business attributes, and the business attribute values ​​corresponding to the business attributes.

[0127] The determining module 304 is further configured to determine the storage location of the log to be extracted in the object storage server based on the log index of the log to be extracted and the first mapping relationship.

[0128] Optionally, the processing module 303 is specifically used to extract the log to be extracted from a target area in the storage location according to the storage level, wherein the target area is the area in the storage location used to store the log corresponding to the storage level.

[0129] Optionally, the processing module 303 is further configured to periodically retrieve log files from the log collector, parse the log files, and obtain the attribute range and storage level corresponding to the log files.

[0130] The generation module 302 is further configured to generate a log index of the log file based on the attribute range corresponding to the log file, and determine the storage location corresponding to the attribute range;

[0131] The processing module 303 is further configured to classify and store the log files to corresponding storage locations in the object storage server according to the storage level corresponding to the log file and the log index, and store the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log file in the database.

[0132] Optionally, the acquisition module 301 is further configured to acquire abnormal logs in the logs to be extracted;

[0133] The processing module 303 is further configured to perform indexing processing on the exception log and store the second mapping relationship between the storage location of the exception log and the log index of the exception log in the database;

[0134] The processing module 303 is also used to perform event analysis processing based on the exception log to obtain analysis results.

[0135] Optionally, the log extraction device based on business attributes further includes: a judgment module 305;

[0136] The judgment module 305 is used to determine whether the analysis results can identify the cause of the anomaly.

[0137] The processing module 303 is further configured to output the anomaly log when the analysis results can determine the cause of the anomaly.

[0138] The processing module 303 is also used to reset the attribute range of the log to be extracted when the analysis result cannot determine the cause of the anomaly, and extract the new abnormal log from the storage location corresponding to the new attribute range.

[0139] The processing module 303 is further configured to perform event analysis processing based on the new anomaly log to obtain new analysis results until the new analysis results can determine the cause of the anomaly.

[0140] Figure 4 A schematic diagram of the log extraction device based on business attributes provided in this application. Figure 4 As shown, this application provides a log extraction device based on business attributes. The log extraction device 400 based on business attributes includes: a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0141] Receiver 401 is used to receive instructions and data;

[0142] Transmitter 402 is used to send commands and data;

[0143] Memory 404 is used to store instructions executed by the computer;

[0144] Processor 403 is used to execute computer execution instructions stored in memory 404 to implement the various steps of the log extraction method based on business attributes in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the log extraction method based on business attributes.

[0145] Alternatively, the memory 404 can be either standalone or integrated with the processor 603.

[0146] When the memory 404 is set up independently, the electronic device also includes a bus for connecting the memory 404 and the processor 403.

[0147] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the business attribute-based log extraction method performed by the aforementioned business attribute-based log extraction device.

[0148] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0149] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0150] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0151] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0152] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0153] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0154] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0155] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0156] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log extraction method based on business attributes, characterized in that, include: Log files are periodically retrieved from the log collector and parsed to obtain the attribute range and storage level of the log files. Based on the attribute range corresponding to the log file, generate the log index of the log file and determine the storage location corresponding to the attribute range; Based on the storage level corresponding to the log file and the log index, the log files are classified and stored in the corresponding storage locations in the object storage server, and the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log file is stored in the database; Obtain the attribute range of the log to be extracted, the attribute range including: query time period, business attribute, and the business attribute value corresponding to the business attribute; Obtain the first mapping relationship stored in the database, which is used to indicate the log index and the storage location of the log in the object storage server; The log index of the log to be extracted is determined based on the query period, business attributes, and the business attribute values ​​corresponding to the business attributes. Based on the log index of the log to be extracted and the first mapping relationship, the storage location of the log to be extracted in the object storage server is determined, and a log extraction request corresponding to the storage location is generated. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges. Based on the storage level, the log to be extracted is extracted from the target area in the storage location, where the target area is the area in the storage location used to store the logs corresponding to the storage level.

2. The method according to claim 1, characterized in that, After retrieving the log to be retrieved from the storage location according to the storage level, the method further includes: Obtain the exception logs from the logs to be extracted; The exception logs are indexed, and the second mapping relationship between the storage location of the exception logs and the log index of the exception logs is stored in the database; Based on the anomaly log, perform event analysis and processing to obtain the analysis results.

3. The method according to claim 2, characterized in that, After performing event analysis processing based on the anomaly log and obtaining the analysis results, the method further includes: Determine whether the analysis results can identify the cause of the anomaly. If so, output the aforementioned exception log; If not, reset the attribute range of the log to be extracted, and extract the new exception log from the storage location corresponding to the new attribute range; Based on the new anomaly log, event analysis processing is performed to obtain new analysis results until the new analysis results can determine the cause of the anomaly.

4. A log extraction device based on business attributes, characterized in that, include: The processing module is used to periodically retrieve log files from the log collector, parse the log files, and obtain the attribute range and storage level of the log files. The generation module is used to generate a log index for the log file based on the attribute range corresponding to the log file, and determine the storage location corresponding to the attribute range; The processing module is further configured to classify and store the log files to corresponding storage locations in the object storage server according to the storage level corresponding to the log files and the log index, and store the first mapping relationship between the storage location corresponding to the attribute range and the log index of the log files in the database; The acquisition module is used to acquire the attribute range of the log to be extracted. The attribute range includes: the query period, the business attribute, and the business attribute value corresponding to the business attribute. The acquisition module is used to acquire a first mapping relationship stored in the database, the first mapping relationship being used to indicate the log index and the storage location of the log in the object storage server; The determination module is used to determine the log index of the log to be extracted based on the query period, business attributes, and the business attribute values ​​corresponding to the business attributes. The determining module is used to determine the storage location of the log to be extracted in the object storage server based on the log index of the log to be extracted and the first mapping relationship; The generation module is used to generate a log extraction request corresponding to the storage location. The log extraction request is used to indicate the storage level of the log to be extracted. Different storage locations store logs with corresponding attribute ranges. The processing module is configured to extract the logs to be extracted from a target area in the storage location according to the storage level, wherein the target area is the area in the storage location used to store logs corresponding to the storage level.

5. A log extraction device based on business attributes, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 3.

Images