Log analysis methods, devices, equipment and storage media

Abstract

This application relates to the field of data processing technology, and discloses a log analysis method, apparatus, device, and storage medium. The method includes: creating a second thread in response to a log selection operation on an interface via a first thread; preprocessing a target log file using the second thread to obtain a first processing result; and calling a third thread in response to a search condition trigger operation on the interface via the first thread to filter out a second processing result matching the search condition from the first processing result, and displaying it on the interface. The log analysis method provided by this application improves log analysis efficiency by processing log files using multiple threads.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a log analysis method, apparatus, device, and storage medium. Background Technology

[0002] In related technologies, log analysis tools struggle to handle compressed files, multiple files, and logs with complex formats such as timestamps, tags, and log levels simultaneously. For example, log files in vehicle systems are typically large, stored as compressed files, and contain multiple files with complex formats. This makes it difficult to perform precise searches within a specified time range, leading to time-consuming problem localization. Summary of the Invention

[0003] This application provides a log analysis method, apparatus, device, and storage medium to solve the problems of low log analysis efficiency and complex operation process.

[0004] In a first aspect, this application provides a log analysis method, which includes: creating a second thread by responding to a log selection operation on an operation interface via a first thread, wherein the log selection operation is the selection of a target log file to be analyzed; performing file preprocessing on the target log file by the second thread to obtain a first processing result, wherein the file preprocessing is used to mark and sort the target log file according to preset rules; and calling a third thread by responding to a search condition trigger operation on an operation interface via the first thread to filter out a second processing result that matches the search condition from the first processing result and display it on the operation interface.

[0005] Based on the aforementioned technical means, this application introduces a multi-threaded collaboration mechanism to separate time-consuming log preprocessing and search filtering tasks from the main thread, effectively improving the overall efficiency of log analysis.

[0006] In one optional implementation, the target log files include one or more. The target log files are preprocessed by a second thread to obtain a first processing result, including: storing each target log file to a preset cache path; scanning each target log file and recording the timestamp of each log entry in each target log file; calculating the corresponding recording time period for each target log file based on the maximum and minimum timestamps in each target log file, and creating a corresponding time series index for each target log file.

[0007] Based on the above technical means, this application optimizes the preprocessing efficiency and time positioning issues in multi-target log file scenarios.

[0008] In one optional implementation, a corresponding time series index is created for each target log file, including: if multiple target log files contain duplicate log records at the same timestamp, the duplicate log records are deleted, and the record time period of the corresponding target log file is updated.

[0009] Based on the aforementioned technical methods, during the creation of a time-series index for each target log file, duplicate log records within the same timestamp in multiple target log files can be effectively identified and deleted, and the affected record time periods can be updated in a timely manner. This fundamentally solves the index redundancy problem caused by duplicate log records.

[0010] In one optional implementation, storing each target log file to a preset cache path includes: when the target log file includes a compressed file, decompressing the compressed file and storing the decompressed file in the cache path.

[0011] Based on the above technical means, by automatically decompressing the compressed file before storing it in the cache path, it is ensured that all target log files, regardless of their original storage format, can exist in the cache path in a uniform, readable uncompressed format.

[0012] In one optional implementation, a first thread responds to a search condition trigger operation on the operation interface, and a third thread is invoked to filter out a second processing result matching the search condition from the first processing result. This includes: the first thread responds to a search condition trigger operation on the operation interface, and the third thread, by invoking the method corresponding to the binary search algorithm, filters out log records associated with the time search in the search condition from the first processing result to obtain a time search result; in the time search result, it searches for log records associated with the tag search in the search condition to obtain tag search results, wherein the tag search includes searching based on multiple parallel tags; and the found tag search results are sorted according to timestamp order to obtain the second processing result.

[0013] Based on the above technical means, the embodiments of this application, through algorithm optimization and step-by-step fine-grained filtering, enable the rapid and accurate acquisition of target log records when faced with complex queries that include time search and multiple tags.

[0014] In one optional implementation, the tag selection process in the tag search is determined by a tag selector, which includes multiple levels. The tag selection process includes: determining the scope of the question to be queried through the first-level tags of the tag selector; determining the specific phenomenon corresponding to the scope of the question to be queried through the second-level tags of the tag selector; and determining the specific question associated with the specific phenomenon through the third-level tags of the tag selector.

[0015] Based on the aforementioned technical means, the progressive tag selection method not only reduces the cost for users to learn complex search syntax and improves the accuracy of problem location, but also, due to its structured nature, enables each analysis process to form a clear search path, facilitating the accumulation and reuse of experience, thereby avoiding repetitive work and improving the overall efficiency of log analysis.

[0016] In one optional implementation, a third thread is invoked to filter out the second processing results that match the search criteria from the first processing results and display them in the user interface. This includes: associating keywords related to the specific question and highlighting the keywords in the second processing results; and adding the tag search records to the historical tag records.

[0017] Based on the aforementioned technical means, by combining intelligent highlighting and historical record management functions, the log analysis process becomes more efficient and convenient, greatly optimizing the user experience. Especially when dealing with complex and massive log data, it can help users diagnose problems faster and reduce repetitive work.

[0018] Secondly, this application provides a log analysis device, comprising: an operation module, configured to create a second thread in response to a log selection operation on an operation interface via a first thread, wherein the log selection operation is the selection of a target log file to be analyzed; a processing module, configured to perform file preprocessing on the target log file via the second thread to obtain a first processing result, wherein the file preprocessing is used to mark and sort the target log file according to preset rules; and a display module, configured to invoke a third thread to filter out a second processing result matching the search conditions from the first processing result in response to a search condition trigger operation on an operation interface via the first thread, and display the second processing result on the operation interface.

[0019] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the log analysis method of the first aspect or any corresponding embodiment described above.

[0020] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the log analysis method of the first aspect or any corresponding embodiment described above.

[0021] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the log analysis method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the first type of log analysis method according to an embodiment of this application; Figure 2 This is a schematic diagram of a second flowchart of the log analysis method according to an embodiment of this application; Figure 3 This is a schematic diagram of the third process of the log analysis method according to the embodiments of this application; Figure 4 This is a structural block diagram of a log analysis device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0027] In related technologies, log analysis methods suffer from low efficiency, difficulty in quickly locating problems, and lack of flexible time-based search mechanisms when processing massive log files, resulting in time-consuming problem location.

[0028] To address this, this application proposes a log analysis method, such as... Figure 1As shown, the method includes: Step S101: Create a second thread by responding to the log selection operation on the operation interface of the first thread. The log selection operation is to select the target log file to be analyzed.

[0029] The first thread can refer to the system's main thread, whose main responsibility is to respond to interactive operations on the user interface and coordinate the creation and management of other auxiliary threads.

[0030] The second thread is created to perform time-consuming background tasks, such as preprocessing large amounts of log files, to avoid blocking the main thread and thus keep the user interface responsive.

[0031] The user interface refers to the graphical user interface through which users interact with the log analysis system. Through this interface, users can select log files, enter search criteria, and view analysis results.

[0032] The log selection operation refers to the user's action of selecting one or more log files to be analyzed by clicking, dragging, or other means on the operation interface.

[0033] The target log file refers to one or more log files that have been selected by the user and designated for subsequent analysis and processing.

[0034] Specifically, when a user selects one or more log files using the file selection controls on the interface, such as clicking the "Select File" button or dragging and dropping files to a specified area, the first thread receives this user interaction event. One implementation approach is for the first thread to directly start a new thread, the second thread, based on the received selection operation, passing the path or reference of the selected target log file as a parameter. Thus, the second thread is assigned the task of processing these files. Another implementation approach is for the first thread to maintain a task queue, encapsulating the log selection operation and its associated target log file information into a task, adding it to the task queue, and having a pre-existing or periodically checked background thread, the second thread, execute the task. In this way, the responsiveness of the main thread is maintained, avoiding potential UI lag when processing a large number of file selections.

[0035] Step S102: The target log file is preprocessed by the second thread to obtain the first processing result. The file preprocessing is used to mark and sort the target log file according to preset rules.

[0036] File preprocessing is a series of preliminary processing steps performed on log files before formal analysis, aiming to optimize data structure and improve the efficiency of subsequent search and analysis.

[0037] The first processing result refers to the set of log data obtained after file preprocessing. The log records in this set are usually marked and sorted according to specific rules, such as recording time.

[0038] The log entries can be marked and sorted according to preset rules, including marking and sorting them by the recording time of the log entries or by the content of the questions.

[0039] Specifically, taking the tagging and sorting of log entries based on their recording time as an example, the second thread, upon receiving the target log file, reads the content of each log file line by line. During the reading process, the second thread identifies the time information contained in each log record. For example, it can extract the date and time strings from the log lines using preset regular expressions or pattern matching algorithms. The extracted time information can be used as a unique identifier for each log record. Subsequently, these time-stamped log records are collected and sorted in simple ascending or descending order according to their recording time. For example, all log records and their timestamps can be stored in a dynamic list, and then a standard sorting algorithm, such as quicksort or mergesort, can be used for overall sorting. After sorting, a set of log records arranged in chronological order is obtained, which is the first processing result. As another implementation, the second thread can compare the timestamp of each log entry with the timestamps of already processed logs and insert it into the correct time position, thereby completing the sorting of logs during the reading process.

[0040] Step S103: The first thread responds to the search conditions triggered by the operation interface, calls the third thread to filter out the second processing result that matches the search conditions from the first processing result, and displays it in the operation interface.

[0041] The third thread is created to perform specific background tasks, such as performing complex search and filtering operations on preprocessed log data, further reducing the burden on the main thread.

[0042] Search criteria triggering an action refers to the behavior of a user entering specific search criteria on the interface and then initiating a log search process by clicking a button or pressing the Enter key.

[0043] Search criteria refer to a set of rules defined by users to filter specific information from log data, such as specified time range, keywords, log levels, etc.

[0044] The second processing result refers to the set of all log records that match the search criteria after filtering by the search criteria in the first processing result.

[0045] Specifically, when a user enters specific search criteria on the interface, such as a time range or a keyword, and clicks the "Search" button, the first thread receives the search criteria trigger and performs an operation. The first thread can then start a new thread, the third thread, passing the user-input search criteria and a reference to the first processing result to it. Upon receiving the search criteria, the third thread iterates through all log records in the first processing result. For each log record, the third thread checks if its timestamp falls within the time range specified by the search criteria, or if its content contains the keyword specified by the search criteria. All matching log records are collected to form the second processing result. For example, the third thread can start from the beginning of the first processing result and check each log record one by one until all records have been checked. Once the second processing result is generated, the third thread can notify the first thread. Upon receiving the notification, the first thread displays the second processing result in the display area of ​​the interface, for example, in a scrollable text box or table. Alternatively, during the filtering process, the third thread can send the matched log records to the first thread in real time, allowing the first thread to gradually update the content displayed on the interface.

[0046] It is understandable that this application effectively improves the overall efficiency of log analysis by introducing a multi-threaded collaboration mechanism to separate time-consuming log preprocessing and search filtering tasks from the main thread. Simultaneously, preprocessing and sorting the target log files based on recording time provides an ordered data foundation for subsequent search operations, thereby enabling flexible time-based searches of log records and solving the problems of low efficiency and inconvenient time-based searches in traditional log analysis.

[0047] In some of the solutions mentioned above in this application, a second thread is proposed to preprocess the target log file to mark and sort the log file according to the recording time. However, when the target log file includes multiple files, this process lacks a unified management mechanism and an efficient time indexing method, resulting in low file processing efficiency and inability to quickly locate the time range of log records, thereby affecting the accuracy of subsequent search and analysis.

[0048] In response, this application further proposes a log analysis method, such as... Figure 2 As shown, the method includes: Step S201: By responding to the log selection operation on the operation interface via the first thread, a second thread is created. The log selection operation involves selecting the target log file to be analyzed. (See details...) Figure 1 Step S101 in the embodiment will not be described again here.

[0049] Step S202 involves preprocessing the target log file using a second thread to obtain the first processing result. Specifically, this includes: Step S2021: Store each target log file to a preset cache path.

[0050] Specifically, storing each target log file in a preset cache path aims to achieve centralized management of target log files and provide a unified and efficient data source for subsequent processing. For example, the system can configure a dedicated local disk directory as the cache path; when a user selects a target log file, the system automatically copies or moves these files to that directory. Alternatively, in a distributed file system, a shared storage area can be specified as the cache path, and target log files can be uploaded to that path via file transfer protocols (such as FTP, SCP) or a distributed file system interface.

[0051] Step S2022: Scan each target log file and record the timestamp of each log entry in each target log file.

[0052] Specifically, scanning each target log file and recording the timestamp of each log entry in that file lays the foundation for subsequent time-based sorting and indexing. For example, the system can read each target log file line by line, using regular expressions or a predefined log format parser to extract the timestamp information from each line and store it in a temporary data structure, such as a list or array. Alternatively, for structured logs, such as JSON or XML formats, the system can directly utilize the corresponding parsing libraries, such as Jackson or Gson, to parse the log entries and obtain the values ​​of the timestamp fields.

[0053] Step S2023: Calculate the corresponding recording time period for each target log file based on the maximum and minimum timestamps in each target log file, and create a corresponding time series index for each target log file.

[0054] Specifically, based on the maximum and minimum timestamps in each target log file, the corresponding recording time period for each target log file is calculated. This aims to quickly understand the time span of each file's content for initial filtering during time-based searches. For example, after completing a timestamp scan, the system can iterate through the list of recorded timestamps, find the minimum and maximum values, and use these two values ​​as the recording time period for that file. Alternatively, while scanning log files, a variable representing the minimum and maximum timestamps of the current file can be maintained in real time. Each time a timestamp is read, it is compared with and updated to the current variable, and the final recording time period is obtained after the scan is complete. A corresponding time-series index is created for each target log file to build an efficient query structure and accelerate time-related search operations. For example, the characteristics of time-series databases can be utilized to import the extracted timestamps and log content into the time-series database, which will automatically manage and optimize the indexing of the time-series data.

[0055] Step S203: The first thread responds to the search conditions triggered by the operation interface, calls the third thread to filter out the second processing result matching the search conditions from the first processing result, and displays it on the operation interface. See details. Figure 1 Step S103 in the embodiment will not be described again here.

[0056] It is understood that, through the above-described technical solution in this embodiment, this application optimizes the preprocessing efficiency and time positioning issues in multi-target log file scenarios. First, by storing each target log file in a preset cache path, centralized management of scattered log files is achieved, avoiding access delays and resource waste caused by inconsistent file locations, and providing a unified and efficient data source for subsequent processing. Second, by scanning each target log file and recording the timestamp of each log entry, accurate time information is provided for each log entry, which is not only the basis for subsequent sorting but also lays the data foundation for accurate time range calculation. Based on this, the corresponding recording time period is calculated according to the maximum and minimum timestamps in each target log file, enabling the system to quickly understand the time span of each file's content. It can determine whether a file might contain logs from a specific time period without in-depth analysis of the file content, greatly improving the efficiency of time positioning. Finally, a corresponding time series index is created for each target log file, constructing an efficient query structure. This allows the system to quickly locate log records within the target time range using the index during time-related search operations, significantly shortening search response time and improving the overall accuracy of the analysis. The above steps work together to solve the problems of low preprocessing efficiency and inaccurate time positioning in scenarios with multiple log files.

[0057] In some of the solutions mentioned above in this application, a corresponding time series index is created for each target log file to improve subsequent search efficiency. However, in this process, when there are duplicate log records with the same timestamp in multiple target log files, it will lead to index redundancy, affecting search accuracy and efficiency.

[0058] In response, this application further proposes creating a corresponding time-series index for each target log file, including: Step a1: If multiple target log files contain duplicate log records at the same timestamp, delete the duplicate log records and update the record time period of the corresponding target log files.

[0059] Specifically, when processing multiple target log files, the system needs to identify whether duplicate log records exist within the same timestamp. The purpose of identifying duplicate log records is to remove redundant data before building the time-series index, ensuring the index's conciseness and accuracy. One implementation is to generate a hash value for each log record, based on both the log content and the timestamp. When encountering log records with the same timestamp, their hash values ​​are compared to determine if they are duplicates. If the hash values ​​are the same, they are considered duplicates. Another implementation is to directly compare all content byte-by-byte or field-by-field for log records with the same timestamp. If all fields are identical, it is determined to be a duplicate. Furthermore, metadata such as log level, source file path, and log message type can be combined for comparison to more accurately identify duplicate log records.

[0060] Once duplicate log records are identified, they need to be deleted. Deleting duplicate log records aims to eliminate redundant data, thereby reducing the total amount of data requiring indexing and searching, improving processing efficiency, and ensuring data consistency. One implementation is to exclude duplicate log records from the index structure during time-series index building, or mark them as "deleted" in the index while leaving the records in the original log files unchanged—this is logical deletion. Another implementation is to remove duplicate log records directly from the processing results in memory or temporary storage during the file preprocessing stage, ensuring that subsequent index building and searching are based solely on unique log data—this is physical deletion. A deduplication storage method can also be used, where, when storing log records in the first processing result—for example, an intermediate database or data structure—only one instance of each unique log record is stored, along with all source files and timestamp information of its occurrence.

[0061] After deleting duplicate log records, the record time period of the corresponding target log file needs to be updated. This is because if the deleted duplicate record happens to correspond to the minimum or maximum timestamp of a target log file, the previously calculated record time period will no longer be accurate. The purpose of updating the record time period is to ensure that the time range covered by the time series index of each target log file precisely matches the time range of the actual unique log records. One implementation is to iterate through all the remaining unique log records in the target log file after deleting duplicate log records, redetermine their minimum and maximum timestamps, and thus obtain the updated record time period. Another implementation is that if the deleted duplicate log record happens to correspond to the minimum or maximum timestamp of a target log file, then the next minimum or maximum timestamp needs to be found as the new boundary, and the record time period needs to be adjusted accordingly. If the deleted record is not at the boundary, no adjustment is needed. A dynamic maintenance method can also be used, where the minimum and maximum timestamps of each target log file are maintained in real time during the construction of the time series index. When a log record is deleted, if the record has the current minimum or maximum timestamp, these two boundary values ​​are updated immediately.

[0062] It is understood that, through the technical solution described in this embodiment, during the creation of a time-series index for each target log file, duplicate log records within the same timestamp in multiple target log files can be effectively identified and deleted, and the affected record time periods can be updated in a timely manner. This fundamentally solves the index redundancy problem caused by duplicate log records, avoids storing and processing unnecessary data in the first processing result, thereby improving the efficiency and accuracy of subsequent search operations. Simultaneously, by updating the record time periods, it ensures that the boundary information of the time-series index always remains consistent with the actual unique log data, avoiding inaccurate time ranges that may result from data deletion, and further enhancing the reliability of log analysis results.

[0063] In some of the solutions mentioned above in this application, the target log file is stored in a preset cache path for file preprocessing. However, in this process, if the target log file is a compressed file, directly storing the compressed file will result in the inability to read the file content, thus making it impossible to perform subsequent scanning and timestamp recording, affecting the effectiveness and efficiency of preprocessing.

[0064] In response, this application further proposes storing each target log file to a preset cache path, including: Step b1: If the target log file includes a compressed file, decompress the compressed file and store the decompressed file in the cache path.

[0065] Specifically, before storing the target log file to the preset cache path, the system first needs to determine whether the target log file is a compressed file. This determination can be achieved in several ways. For example, it can check the file extension of the target log file, such as .zip, .tar.gz, .rar, etc., to identify whether it is a known compression format; or it can read the header information of the target log file and determine the compression type of the file by identifying specific byte sequences. Once the target log file is determined to be a compressed file, the system will perform a decompression operation. The decompression operation can be completed using the standard decompression library or API interface provided by the operating system, such as calling the java.util.zip library or Python's zipfile module; or it can be completed by calling external decompression tools, such as unzip, tar, and other command-line tools. After decompression is completed, the resulting decompressed file, i.e., the original log content, will be stored in the preset cache path. The storage procedure can directly write the decompressed data to a new file under the cache path, or it can temporarily store the decompressed data in a memory buffer and then write it to the cache path.

[0066] It is understood that, through the above-described technical solution in this embodiment, this application solves the problem that subsequent preprocessing cannot be directly performed when the target log file is in compressed format. By automatically decompressing the compressed file before storing it in the cache path, it ensures that all target log files, regardless of their original storage format, exist in the cache path in a uniform, readable uncompressed format. This allows subsequent preprocessing steps such as log file scanning, timestamp recording, and time series index creation to proceed smoothly, avoiding processing interruptions or data omissions caused by file format incompatibility. This application can guarantee the integrity and accuracy of the first processing result, significantly improving the compatibility and processing efficiency of log analysis methods for complex log file formats, thereby providing a reliable data foundation for subsequent log search and analysis.

[0067] In some of the solutions described above in this application, a third thread is used to filter out the second processing result that matches the search criteria from the first processing result to quickly locate log records. However, in this process, when the search criteria include time search and multiple parallel tags, this application further proposes an optimized log filtering method, such as... Figure 3 As shown, the method includes: Step S301: By responding to the log selection operation on the operation interface via the first thread, a second thread is created. The log selection operation involves selecting the target log file to be analyzed. (See details...) Figure 1 Step S101 in the embodiment will not be described again here.

[0068] Step S302 involves preprocessing the target log file using a second thread to obtain the first processing result. Specifically, this includes: Step S3021: Store each target log file to a preset cache path. See details... Figure 2 Step S2021 in the embodiment will not be repeated here.

[0069] Step S3022: Scan each target log file and record the timestamp of each log entry in each target log file. See details... Figure 2 Step S2022 in the embodiment will not be repeated here.

[0070] Step S3023: Based on the maximum and minimum timestamps in each target log file, calculate the corresponding recording time period for each target log file, and create a corresponding time series index for each target log file. (See details...) Figure 2 Step S2023 in the embodiment will not be repeated here.

[0071] Step S303 involves the first thread responding to the search criteria triggered by the operation interface, and then calling the third thread to filter out the second processing result matching the search criteria from the first processing result. Specifically, this includes: In step S3031, the first thread responds to the search conditions triggered by the operation interface, and the third thread calls the method corresponding to the binary search algorithm to filter out the log records related to the time search in the first processing result, and obtain the time search result.

[0072] Specifically, the third thread calls the method corresponding to the binary search algorithm to filter out log records associated with the time search criteria from the first processing result, thus obtaining the time search result. This method aims to efficiently locate and extract log records that meet a specific time range from the first processing result, which has already been sorted by time. Since the first processing result has been preprocessed and marked and sorted according to preset rules, its ordered nature can be fully utilized. One implementation is that after receiving the time search criteria, such as a start time point and an end time point, the third thread first performs two binary searches on the timestamp sequence in the first processing result. The first search locates the index of the first log record greater than or equal to the start time point, and the second search locates the index of the first log record greater than the end time point. All log records between these two indices constitute the time search result. Another implementation is that the third thread can use an interpolation search algorithm or a Fibonacci search algorithm. When the log timestamp distribution is relatively uniform, the interpolation search algorithm can converge to the target position faster than the binary search algorithm. When the log data volume is extremely large and memory is limited, a disk-based binary search variant can be used to filter the time range without fully loading the data.

[0073] In one example, time search can support multiple input formats, such as HH:MM:SS, HHMMSS, HH:MMSS, etc., as well as automatically extracting numerical information from the input and combining them with the time format.

[0074] Step S3032: In the time search results, find the log records associated with the tag search in the search criteria to obtain the tag search results, wherein the tag search includes searching based on multiple parallel tags.

[0075] Specifically, this step is used to further refine the filtering of the log record set, which has already been narrowed down by time search (i.e., the time search results), based on multiple tags specified by the user. Here, "parallel" means that the user can specify multiple tags simultaneously, and these tags can have logical relationships, such as "AND" or "OR". One implementation is that a third thread iterates through each log record in the time search results, checking if the content of each record contains all the specified parallel tags. For example, if the user specifies tag A and tag B, only log records containing both tags A and B will be included in the tag search results. Another implementation is that a third thread iterates through each log record in the time search results, checking if its content contains either one of the specified parallel tags. For example, if the user specifies tag A or tag B, any log record containing either tag A or tag B will be included in the tag search results. Furthermore, more complex Boolean logic combinations can be supported, such as "(tag A AND tag B) OR tag C", by parsing the logical expression entered by the user to perform the corresponding matching operation.

[0076] Step S3033: Sort the found tag search results according to the timestamp order to obtain the second processing result.

[0077] Specifically, this step aims to ensure that the final processed results presented to the user maintain temporal continuity and readability, even if the relative order of log records may change due to matching logic during the tag search process. One implementation involves a third thread comparing the timestamps of these log records after completing the tag search and collecting all matching log records, and then rearranging them into strict ascending chronological order using standard sorting algorithms such as quicksort, mergesort, or heapsort. Another implementation approach is to use algorithms more suitable for partially ordered data, such as insertion sort or bucket sort, if the relative chronological order of log records can be maintained during the tag search process—for example, by directly inserting matching log records into a sorted list while iterating through the time search results—to reduce the computational overhead of overall sorting. For instance, a small ordered list can be maintained for each timestamp range, and these small lists can then be merged.

[0078] It is understood that, through the above technical solution, this application effectively solves the problem of low log filtering efficiency under complex search conditions. Firstly, by having a first thread respond to the search conditions triggered by the user interface and then calling a third thread for processing, multi-threaded collaboration is achieved, improving system response speed. The third thread utilizes the binary search algorithm to quickly filter out log records related to the time search from the first processing result, fully leveraging the temporal order of the first processing result and avoiding a full scan of massive log data, significantly improving the efficiency and accuracy of time range positioning. Based on this, tag search is performed within the narrowed time search results, further reducing the amount of data searched. By supporting searches with multiple parallel tags, the flexibility and expressiveness of the query are enhanced, enabling users to accurately locate problems based on more complex business logic. Finally, the found tag search results are sorted according to timestamp order, ensuring that the final second processing result displayed to the user is clear, orderly, and easy to analyze, improving user experience and problem-solving efficiency. This application's embodiments, through algorithm optimization and step-by-step refined filtering, enable the rapid and accurate acquisition of target log records when faced with complex queries involving time searches and multiple tags, thereby effectively improving the efficiency of log analysis and user satisfaction.

[0079] In some of the solutions mentioned above in this application, tag search is proposed to filter log records. However, in the process of implementation, users may find it difficult to efficiently specify the scope of the problem, the specific phenomenon and the specific problem, resulting in high learning costs, inaccurate positioning and inability to accumulate experience.

[0080] In this regard, this application further proposes that the tag selection in the tag search process is determined by a tag selector, which includes multiple levels, and the tag selection process includes: Step c1: Use the first-level tags of the tag selector to determine the scope of the question to be queried; use the second-level tags of the tag selector to determine the specific phenomenon corresponding to the scope of the question to be queried; use the third-level tags of the tag selector to determine the specific problem associated with the specific phenomenon.

[0081] Specifically, a tag selector is an interface or mechanism used to assist users in selecting tags in a structured and hierarchical manner. It aims to break down complex search criteria into easily understandable and operable hierarchical structures. Tag selectors can manifest as dropdown menus, tree structures, or cascading selection boxes in a graphical user interface (GUI), allowing users to progressively narrow their search by clicking or selecting predefined tags. Furthermore, tag selectors can also be intelligent suggestion systems based on text input, automatically recommending tag options that conform to the hierarchical structure when the user enters partial keywords. Tag selectors can be multi-level, meaning they contain at least two levels of tags, each representing a different granularity of information classification, with lower-level tags being refinements or subordinate relationships to higher-level tags. These tag levels can be pre-configured and maintained by system administrators or domain experts based on the characteristics of log data and common problem types, or they can be dynamically generated and optimized using machine learning algorithms based on historical log data and user query patterns.

[0082] In the tag selection process, the first level of the tag selector first defines the scope of the problem to be queried. The first level tags define the problem area of ​​concern to the user from a macro perspective, thus quickly narrowing the initial search scope. For example, in an in-vehicle system, the first level tags could be broad categories such as "system failure," "performance anomaly," or "user operation error," or classifications based on system architecture such as "module A problem," "module B problem," or "network problem." Based on this, the second level tags of the tag selector determine the specific phenomena corresponding to the scope of the problem to be queried. These second level tags further refine the problem within the scope defined by the first level tags, focusing on specific phenomena or symptoms. For example, if the first level tag is "system failure," the second level tags could be "startup failure," "system crash," or "functional abnormality"; if the first level tag is "performance anomaly," the second level tags could be "response delay," "high resource consumption," or "data transmission interruption." Finally, the third level tags of the tag selector determine the specific problem associated with the specific phenomenon. These third level tags, under the specific phenomenon determined by the second level tags, pinpoint the root cause or specific problem point leading to the phenomenon. For example, if the second-level label is "startup failed", the third-level label could be "configuration file error", "dependent service not started", or "hardware initialization failed"; if the second-level label is "response delay", the third-level label could be "slow database query", "insufficient network bandwidth", or "high algorithm complexity".

[0083] It is understood that, through the above-described technical solution in this embodiment, this application effectively solves the problems of users' difficulty in efficiently specifying the scope of issues, specific phenomena, and specific problems during log analysis, as well as the resulting high learning costs, inaccurate positioning, and inability to accumulate experience. Specifically, the multi-level tag selector provides a structured and hierarchical search guidance mechanism. Users first determine the scope of the problem to be queried from a macro perspective using first-level tags, quickly narrowing the initial search space and reducing the user's cognitive burden. Next, within the scope determined by the first-level tags, users further determine the specific phenomena corresponding to this scope using second-level tags, guiding the search focus to more specific events or symptoms. Finally, users accurately determine the specific problems associated with the specific phenomena using third-level tags, making the positioning of log records more precise. This progressive tag selection method not only reduces the cost for users to learn complex search syntax and improves the accuracy of problem positioning, but also, due to its structured nature, allows each analysis process to form a clear search path, facilitating the accumulation and reuse of experience, thereby avoiding repetitive work and improving the overall efficiency of log analysis.

[0084] In some of the solutions mentioned above in this application, a third thread is proposed to filter out the second processing result that matches the search criteria from the first processing result and display it in the operation interface to quickly present the search results. However, in this process, it is difficult for users to quickly locate key information in a large number of log records and the analysis history cannot be saved, which leads to the need to repeatedly set the label conditions in subsequent analyses, increasing the operational burden and repetitive work.

[0085] In response, this application further proposes calling a third thread to filter out the second processing result matching the search criteria from the first processing result, and displaying it in the operation interface, including: Step d1: Associate keywords with specific questions and highlight them in the second processing result.

[0086] Specifically, this step aims to address the problem of users struggling to quickly locate key information within a large volume of log records. Visual highlighting effectively guides user attention and improves information retrieval efficiency. For example, the system can pre-configure or allow administrators to configure a keyword library that maps to the third-level tags in the tag selector—specific questions. When a user selects a specific question through the tag selector, the system automatically loads and matches the corresponding keywords against the log content in the second processing result, highlighting the matched text fragments with different colors, fonts, or background colors. Furthermore, users can also define their own keywords related to the current specific question. Users can manually enter or select keywords in the search interface, and the system uses these user-defined keywords along with pre-defined keywords for highlighting. Highlighting methods can include, but are not limited to, changing text color, bolding, underlining, and background color filling to enhance visual recognition.

[0087] Step d2: Add the tag search records to the historical tag records.

[0088] Specifically, this step aims to address the problem of failing to save analysis history, leading to repetitive work. By recording users' search criteria, it allows for quick review and reuse of historical searches, accumulating analytical experience. For example, each time a user completes a tag search and triggers the results display, the system can automatically package all tags used in that search, including tags at all levels of the tag selector, as well as other search criteria such as time ranges and keywords, into a single search record entry and store it in local storage or a server-side history database. These records can be arranged chronologically and offer management functions such as name editing and deletion. Alternatively, a "Save Search" button can be added to the user interface, allowing users to actively click this button after performing a tag search to save the current tag combination and search criteria as a named record. These records can be categorized and managed, and support one-click loading for quick reuse in subsequent analyses. The system can also provide an automatic saving function for recent search records, eliminating the need for manual user intervention.

[0089] It is understood that, through the above-described technical solution in this embodiment, this application effectively solves the problems of users finding it difficult to quickly locate key information and save analysis history during log analysis. By highlighting keywords associated with specific issues in the second processing result, users can immediately identify entries related to the core issue when browsing filtered log records, avoiding the time-consuming manual search in massive amounts of text, thereby significantly improving the efficiency and accuracy of issue location. Simultaneously, by adding tag search records to historical tag records, the system can automatically save the tag search conditions set by the user, facilitating direct reuse later, reducing repetitive input and condition setting operations, and supporting the continuous accumulation of analysis experience. The combination of intelligent highlighting and historical record management functions makes the log analysis process more efficient and convenient, greatly optimizing the user experience, especially when dealing with complex and massive log data, helping users diagnose problems faster and reduce repetitive work.

[0090] In some optional embodiments, the system can generate performance reports based on the content of log analysis, such as extracting timestamps and messages from filtered logs, analyzing the time intervals between key events, identifying abnormally time-consuming operations, and generating performance reports.

[0091] In one example, a more specific example will be used to illustrate the above technical solution in greater detail, as follows: In a vehicle system fault diagnosis scenario, User A needs to analyze massive amounts of log files generated by an onboard system, such as the JK project, to locate an intermittent engine malfunction. These log files are typically stored as compressed packages containing multiple sub-files with complex formats, making them difficult to handle efficiently using traditional manual search or general text processing tools.

[0092] First, user A selects a batch of target log files to be analyzed through the log analysis tool's interface. These files may include multiple compressed files and uncompressed log files. The first thread in the system responds to user A's log selection operation and then creates a second thread.

[0093] The second thread begins preprocessing these target log files. If a target log file contains compressed files, the second thread first decompresses these compressed files and stores the decompressed files in a preset cache path. Next, the second thread stores each target log file in this cache path. Subsequently, the second thread scans each target log file, recording the timestamp of each log entry. Based on the maximum and minimum timestamps in each target log file, the second thread calculates the corresponding recording time period for each target log file and creates a corresponding time series index for each target log file. During this process, if duplicate log records with the same timestamp are found in multiple target log files, the second thread deletes them and updates the recording time period of the corresponding target log files. Through this series of preprocessing steps, all log files are marked and sorted according to preset rules, generating the first processing result. Compared to traditional manual decompression, file-by-file searching, or using general text tools to handle complex formats, the multi-threaded, automated preprocessing method improves efficiency and can effectively handle the complex formats of compressed packages and multi-file logs.

[0094] After preprocessing, User A entered search criteria on the interface, such as specifying a time range, like "yesterday afternoon 2 pm to 3 pm," and a question tag, like "engine failure." User A triggered the search criteria trigger operation, the first thread responded to this operation, and then called the third thread.

[0095] The third thread first calls the corresponding method of the binary search algorithm to quickly filter out log records that match the time range specified by user A from the first processing result, thus obtaining the time search result. Based on the mechanism of time series indexing and binary search, it is more efficient and accurate than traditional line-by-line scanning or simple time filtering, solving the problem of lacking a flexible time search mechanism.

[0096] Next, the third thread further searches the time-based search results for log records related to the tag search criteria entered by user A. User A determined the search tags using a multi-level tag selector. For example, user A first selected "vehicle system" using the first-level tag selector, then "powertrain system" using the second-level tag selector, and finally determined the specific problem as "engine failure" using the third-level tag selector. The multi-level tag selector helps user A accurately pinpoint the problem area without needing to memorize complex keywords or commands, solving the problems of lack of scenario adaptability and high learning costs. The third thread then performs the search based on these tags to obtain the tag search results.

[0097] After obtaining the tag search results, the third thread sorts the results by timestamp to generate the final second processing result. To further assist user A's analysis, the system associates keywords related to the specific issue of "engine failure," such as "overheating" and "sensor malfunction," and highlights these keywords in the second processing result. Finally, the system displays these second processing results to user A in the user interface. Simultaneously, to facilitate user A's subsequent review or similar analysis, the tag search record "vehicle system" -> "powertrain system" -> "engine failure" is added to the historical tag record, resolving the issue of not being able to retain analysis history.

[0098] This embodiment also provides a flag analysis device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] This embodiment provides a character analysis device, such as... Figure 4 As shown, it includes: The operation module 401 is used to create a second thread by responding to the log selection operation on the operation interface through the first thread. The log selection operation is to select the target log file to be analyzed.

[0100] The processing module 402 is used to perform file preprocessing on the target log file through the second thread to obtain the first processing result. The file preprocessing is used to mark and sort the target log file according to preset rules.

[0101] The display module 403 is used to respond to the search conditions triggered by the operation interface through the first thread, call the third thread to filter out the second processing result that matches the search conditions from the first processing result, and display it in the operation interface.

[0102] In some alternative implementations, the processing module 402 includes: The first unit is used to store each target log file to a preset cache path.

[0103] The second unit is used to scan each target log file and record the timestamp of each log entry in each target log file.

[0104] The third unit is used to calculate the corresponding recording time period for each target log file based on the maximum and minimum timestamps in each target log file, and to create a corresponding time series index for each target log file.

[0105] In some alternative implementations, the display module 403 includes: The fourth unit is used to trigger operations by responding to the search conditions of the operation interface through the first thread. The third thread, by calling the method corresponding to the binary search algorithm, filters out the log records related to the time search in the first processing result to obtain the time search result.

[0106] The fifth unit is used to find log records associated with the tag search in the search criteria in the time search results, and obtain the tag search results, wherein the tag search includes searching based on multiple parallel tags; The sixth unit is used to sort the search results for the found tags according to the timestamp order, and obtain the second processing result.

[0107] The log analysis apparatus provided in this application can execute the log analysis method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0108] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0109] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0111] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the log analysis method of embodiments of this application.

[0112] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0113] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the log analysis method shown in the above embodiments is implemented.

[0114] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0115] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A log analysis method, characterized in that, The method includes: A second thread is created by responding to the log selection operation on the operation interface in the first thread. The log selection operation is the selection of the target log file to be analyzed. The target log file is preprocessed by the second thread to obtain a first processing result, wherein the file preprocessing is used to mark and sort the target log file according to a preset rule; The first thread responds to the search conditions triggered by the operation interface, calls the third thread to filter out the second processing result that matches the search conditions from the first processing result, and displays it on the operation interface.

2. The method according to claim 1, characterized in that, The target log files include one or more, and the preprocessing of the target log files by the second thread to obtain a first processing result includes: Each of the target log files is stored in a preset cache path; Scan each of the target log files and record the timestamp of each log entry in each of the target log files; Based on the maximum and minimum timestamps in each target log file, the corresponding recording time period for each target log file is calculated, and a corresponding time series index is created for each target log file.

3. The method according to claim 2, characterized in that, Creating a corresponding time-series index for each of the target log files includes: If multiple target log files contain duplicate log records at the same timestamp, the duplicate log records are deleted, and the corresponding record time period in the target log files is updated.

4. The method according to claim 2, characterized in that, The step of storing each of the target log files to a preset cache path includes: When the target log file includes a compressed file, the compressed file is decompressed, and the decompressed file is stored in the cache path.

5. The method according to claim 1, characterized in that, The step of triggering a search condition on the user interface via a first thread, and then calling a third thread to filter out a second processing result matching the search condition from the first processing result, includes: The first thread responds to the search conditions triggered by the operation interface, and the third thread, by calling the method corresponding to the binary search algorithm, filters out the log records associated with the time search in the first processing result to obtain the time search result. In the time search results, log records associated with the tag search in the search criteria are found to obtain tag search results, wherein the tag search includes searching based on multiple parallel tags; The search results for the identified tags are sorted according to their timestamps to obtain the second processing result.

6. The method according to claim 5, characterized in that, The tag selection process in the tag search is determined by a tag selector, which includes multiple levels. The tag selection process includes: The scope of the query is determined by the first-level tags of the tag selector; The specific phenomena corresponding to the range of questions to be queried are determined by using the second-level tags of the tag selector. The specific problem associated with the specific phenomenon is determined by using the third-level label of the label selector.

7. The method according to claim 6, characterized in that, The step of calling a third thread to filter out a second processing result that matches the search criteria from the first processing result and displaying it on the operation interface includes: Associate keywords related to the specific problem and highlight those keywords in the second processing result; Add the tag search record to the historical tag record.

8. A log analysis device, characterized in that, The device includes: The operation module is used to create a second thread by responding to the log selection operation of the operation interface through the first thread. The log selection operation is the selection of the target log file to be analyzed. The processing module is used to perform file preprocessing on the target log file through the second thread to obtain a first processing result, wherein the file preprocessing is used to mark and sort the target log file according to a preset rule; The display module is used to respond to the search conditions triggered by the operation interface through the first thread, call the third thread to filter out the second processing result that matches the search conditions from the first processing result, and display it in the operation interface.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the log analysis method of any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the log analysis method according to any one of claims 1 to 7.

Images