Log retrieval method, log retrieval device, electronic equipment and storage medium
By segmenting log files and generating summaries through large language model analysis, the problem of low retrieval efficiency caused by log file storage pressure is solved, achieving efficient log retrieval and reduced storage costs.
Patent Information
- Application Number
- CN202511110310.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-08
- Publication Date
- 2025-11-21
AI Technical Summary
A large number of log files increased the storage pressure on the business system, resulting in low log retrieval efficiency.
By acquiring log files and log formats, logs are segmented into log shards. A pre-trained large language model is used for log analysis and summary generation, generating shard summaries for the log shards and log summaries for the log files. The log files are then archived to reduce storage costs.
It improves the efficiency of log retrieval, reduces storage costs, and accelerates the retrieval of archived logs through log summaries.
Smart Images
Figure CN120994616A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and is applied to the fields of financial technology and digital medicine, and in particular relates to a log retrieval method, a log retrieval device, an electronic device and a storage medium. BACKGROUND
[0002] Log files record various events that occur when a system is running, including normal operations and abnormal situations. With the development of information technology, the data processing capacity of business systems has gradually increased, resulting in a large number of log files. For example, in the financial technology scenario, a bank business system needs to process a large number of transactions every day. In order to be able to discover abnormal transaction behavior in a timely manner, the bank business system will record detailed information of each transaction, such as transaction time, amount, and account information of both parties of the transaction, resulting in a large number of transaction log files. For example, in the digital medical scenario, in order to fully understand the health status of patients and provide high-quality medical services, a medical system will record information such as the medical history, diagnosis results, and medical plans of patients, resulting in a large number of medical log files.
[0003] A large number of log files will increase the storage pressure of the business system. In order to reduce the storage pressure of the business system, the log files will be compressed and archived to a storage medium with lower cost, resulting in low efficiency of log retrieval. SUMMARY
[0004] The main purpose of the embodiments of the present application is to propose a log retrieval method, a log retrieval device, an electronic device and a storage medium, which aims to improve the efficiency of log retrieval.
[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes a log retrieval method, which comprises:
[0006] obtaining a log file and a log format of the log file;
[0007] segmenting the log file to obtain log segments;
[0008] performing log analysis on the log format, the log segments and a preset segment prompt instruction of the log segments by a pre-trained large language model to obtain a segment summary of the log segments;
[0009] generating a summary by the large language model on the log format, the segment summary and a preset summary prompt instruction to obtain a log summary of the log file;
[0010] archiving the log file to obtain an archived log;
[0011] performing log retrieval on the archived log according to the log summary.
[0012] In some embodiments, the log analysis on the log format, the log shard and the preset shard prompt instruction by the pre-trained large language model obtains a shard summary of the log shard, including:
[0013] obtaining a thread concurrency degree, and starting a target thread according to the thread concurrency degree;
[0014] obtaining the log shard by the target thread to obtain a reference shard;
[0015] screening a preset shard prompt instruction according to the reference shard to obtain a reference prompt instruction;
[0016] calling the large language model by the target thread, and obtaining the shard summary by the called large language model on the log format, the reference shard and the reference prompt instruction.
[0017] In some embodiments, the obtaining of the thread concurrency degree includes:
[0018] obtaining a number of shards of the log shard;
[0019] generating a shard summary according to the number of shards and a preset summary generation model to obtain a shard summary generation time length;
[0020] determining a target time length according to the shard summary generation time length and the number of shards;
[0021] calculating the thread concurrency degree according to the target time length and a preset expected time length.
[0022] In some embodiments, the summary generation model includes an encoder and a decoder, and the generating of the shard summary according to the number of shards and the preset summary generation model includes:
[0023] screening the log shard according to the number of shards to obtain selected shards;
[0024] performing feature encoding on the selected shards by the encoder to obtain log encoding features;
[0025] performing feature decoding on the log encoding features by the decoder to obtain a candidate summary of the selected shards, and obtaining a generation time length of the candidate summary to obtain a candidate time length;
[0026] calculating the shard summary generation time length according to the candidate time length.
[0027] In some embodiments, the feature encoding of the selected slice by the encoder obtains log encoding features, including:
[0028] The vectorization processing of the selected slice by the encoder obtains a slice feature vector;
[0029] The attention calculation of the slice feature vector obtains a slice attention feature;
[0030] The feature extraction of the slice attention feature obtains the log encoding features.
[0031] In some embodiments, the calculation of the thread concurrency according to the target duration and the preset expected duration includes:
[0032] According to the target duration and the preset expected duration, an initial concurrency is calculated;
[0033] Obtain system load data;
[0034] According to the system load data, the initial concurrency is updated to obtain the thread concurrency.
[0035] In some embodiments, the archiving of the log file obtains an archived log, including:
[0036] Obtain the file generation duration of the log file;
[0037] According to the file generation duration, determine the archiving parameters of the log file;
[0038] According to the archiving parameters, the log file is archived to obtain the archived log.
[0039] To achieve the above purpose, a second aspect of the embodiment of the present application proposes a log retrieval device, the device includes:
[0040] The acquisition module is configured to acquire a log file and a log format of the log file;
[0041] The segmentation module is configured to segment the log file to obtain log slices;
[0042] The log analysis module is configured to perform log analysis on the log format, the log slices, and a preset slice prompt instruction of the log slices by a pre-trained large language model to obtain a slice summary of the log slices;
[0043] The summary generation module is configured to perform summary generation on the log format, the slice summary, and a preset summary prompt instruction by the large language model to obtain a log summary of the log file;
[0044] An archiving module is configured to archive the log file to obtain an archived log;
[0045] A log searching module is configured to search the archived log according to the log summary.
[0046] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.
[0047] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.
[0048] The log searching method, the log searching device, the electronic device and the computer readable storage medium provided by the embodiments of the present application can obtain a log file and a log format of the log file, so as to understand the log file based on the log format. In order to make the log file meet the model input requirement, the log file is segmented to obtain log fragments. In order to generate a summary meeting the business requirement, a summary prompt instruction is constructed, and a pre-trained large language model is used to analyze the log format, the log fragments and a preset fragment prompt instruction of the log fragments to obtain a fragment summary of the log fragments. The large language model is used to generate a summary based on the log format, the fragment summary and the preset summary prompt instruction, so as to summarize summaries of multiple log fragments to obtain a log summary of the log file. In order to reduce the storage cost of the log file, the log file is archived to obtain an archived log. The archived log is searched according to the log summary, so as to improve the efficiency of log searching. BRIEF DESCRIPTION OF DRAWINGS
[0049] Figure 1 is a flowchart of the log searching method provided by the embodiments of the present application;
[0050] Figure 2 is a flowchart of step S130 in Figure 1
[0051] Figure 3 is a flowchart of step S210 in Figure 2
[0052] Figure 4 is a flowchart of step S320 in Figure 3
[0053] Figure 5 is a flowchart of step S420 in Figure 4
[0054] Figure 6 is a flowchart of step S340 in Figure 3
[0055] Figure 7 is a flowchart of step S150 in Figure 1
[0056] Figure 8 is a structural schematic diagram of a log retrieval device provided by an embodiment of the present application;
[0057] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0059] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.
[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0061] Log files record various events that occur during system operation, including normal operation and abnormal situations. With the development of information technology, the data processing capacity of business systems is gradually enhanced, resulting in a large number of log files. For example, in the financial technology scenario, a bank business system needs to process a large number of transactions every day. In order to be able to discover abnormal transaction behavior in a timely manner, the bank business system will record detailed information of each transaction, such as transaction time, amount, and transaction account information of both parties, resulting in a large number of transaction log files. For example, in the digital medical scenario, in order to fully understand the health status of patients and provide high-quality medical services, a medical system will record information such as patient medical history, diagnosis results, and medical plans, resulting in a large number of medical log files.
[0062] A large number of log files will increase the storage pressure of the business system. In order to reduce the storage pressure of the business system, the log files will be compressed and archived to a storage medium with lower cost, resulting in low efficiency of log retrieval.
[0063] Based on this, the log retrieval method, log retrieval device, electronic device and computer readable storage medium provided by the embodiments of the present application aim to improve the efficiency of log retrieval.
[0064] The log retrieval method, log retrieval device, electronic device and computer readable storage medium provided by the embodiments of the present application are specifically described by the following embodiments. First, the log retrieval method in the embodiments of the present application is described.
[0065] The log retrieval method provided by the embodiments of the present application relates to the technical field of artificial intelligence. The log retrieval method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application that implements the log retrieval method, but is not limited to the above forms.
[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.
[0067] Figure 1 is an optional flowchart of the log retrieval method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps S110 to S160.
[0068] Step S110, obtaining a log file and a log format of the log file;
[0069] Step S120, the log file is cut to obtain a log fragment;
[0070] Step S130, the log format, the log fragment and the preset fragment prompt instruction of the log fragment are analyzed by the pre-trained large language model to obtain a fragment summary of the log fragment;
[0071] Step S140, the log format, the fragment summary and the preset summary prompt instruction are generated by the large language model to obtain a log summary of the log file;
[0072] Step S150, the log file is archived to obtain an archived log;
[0073] Step S160, the archived log is searched according to the log summary.
[0074] The steps S110 to S160 shown in the embodiments of the application are to obtain the log file and the log format of the log file, so as to understand the log file based on the log format. In order to make the log file meet the model input requirements, the log file is cut to obtain a log fragment. In order to generate a summary that meets the business requirements, a summary prompt instruction is constructed, and the log format, the log fragment and the preset fragment prompt instruction of the log fragment are analyzed by the pre-trained large language model to obtain a fragment summary of the log fragment. The log format, the fragment summary and the preset summary prompt instruction are generated by the large language model to summarize the summaries of multiple log fragments and obtain a log summary of the log file. In order to reduce the storage cost of the log file, the log file is archived to obtain an archived log. The archived log is searched according to the log summary, which improves the efficiency of log searching.
[0075] In step S110 of some embodiments, a log file generated in the running process of a system or an application is obtained, and the log file includes different types of logs, such as system logs, application logs and security logs. The log file can be exported through an event viewer of a file system, can be obtained from a specified path of the file system, or can be obtained through an interface call. The log format of the log file is also obtained, which defines the format of log record generation. For example, the log format for website access includes client address, access time, response status code, etc. In order to enable the large language model to understand the meaning of the log entry and realize automatic identification of log semantics, the file name of the log file can be stored as a key and the log format of the log file as a value.
[0076] In step S120 of some embodiments, there is a certain degree of limitation on the model input length. In order to make the length of the log file meet the model input requirement, the model input length is obtained, which can be 500 characters. The log file is divided according to the model input length, and a plurality of log fragments are obtained.
[0077] Please refer to Figure 2 In some embodiments, step S130 can include but is not limited to steps S210 to S240:
[0078] In step S210, the thread concurrency is obtained, and the target thread is started according to the thread concurrency.
[0079] In step S220, the log fragment is obtained by the target thread, and the reference fragment is obtained.
[0080] In step S230, the reference fragment is used to filter the preset fragment prompt instruction, and the reference prompt instruction is obtained.
[0081] In step S240, the large language model is called by the target thread, and the log format, the reference fragment and the reference prompt instruction are analyzed by the called large language model to obtain the fragment summary.
[0082] In step S210 of some embodiments, the thread concurrency is obtained, which is the number of threads that can run simultaneously. The target thread is started according to the thread concurrency. The target thread is used to execute the summary generation task of the log fragment, and the number of target threads is equal to the thread concurrency.
[0083] In step S220 of some embodiments, multiple threads executing the same task may cause resource competition, data inconsistency and other problems. In order to improve the summary generation efficiency of the log file, the summary generation task of the log fragment is distributed to the target thread, and the corresponding log fragment is obtained by the target thread to obtain the reference fragment. Different target threads cannot access the same log fragment.
[0084] In step S230 of some embodiments, a preset shard prompt instruction can be set for each log shard, or the same preset shard prompt instruction can be set for multiple log shards. The preset shard prompt instruction is used to prompt the large language model to generate a summary for the log shard, and can be defined in advance before performing the summary generation task. For example, the preset shard prompt instruction can be: "which methods and interfaces are accessed frequently; which interfaces have a time consumption of more than 1 second; when is the peak of interface requests, and the top 10 can be taken; what are the access trends and response trends of interfaces and methods; which abnormal error keywords are worth paying attention to". In the case that the preset shard prompt instructions of the log shards are different, the preset shard prompt instruction of the reference shard can be selected from the preset shard prompt instructions of the multiple log shards to obtain a reference prompt instruction. In the case that the preset shard prompt instructions of the log shards are the same, the preset shard prompt instruction can be used as the reference prompt instruction.
[0085] In step S240 of some embodiments, the large language model is called by the target thread. The large language model is a large natural language model pre-trained by a large data set, and has strong semantic understanding and logical reasoning capabilities. The log format, the reference shard, and the reference prompt instruction are input into the called large language model for log analysis to generate a shard summary of the log shard. The log format and the reference prompt instruction can be used as context to guide the summary generation process of the large language model, wherein the log format is used to make the large language model understand the semantics of the reference shard, and the reference prompt instruction is used to provide the large language model with summary generation requirements to constrain the large language model to generate a summary that meets the requirements. The log format is automatically identified by the large language model, which improves the universality of the program and eliminates the need to develop different identification logics for different log formats.
[0086] The large language model includes a first model and a second model. The first model is used to generate a shard summary of the log shard, and the second model is used to generate a log summary of the log file. The first model and the second model can be the same or different. In order to improve the accuracy of summary generation, expert scores of multiple LLM models can be obtained. The expert scores are used to evaluate the response speed of the LLM model. The LLM model with a high expert score, i.e., the LLM model that can quickly respond to model input, is selected as the first model. The LLM model with a better logical reasoning performance is selected as the second model.
[0087] Considering the security of production operation data such as financial business data (e.g., account balance, transaction flow), medical data (e.g., patient case, diagnosis and treatment record), etc., the large language model is deployed locally and privately in the embodiments of the present application, rather than calling an external LLM service.
[0088] To avoid the hallucination problem of the large language model, the RAG technology can be added, which is a retrieval-augmented generation technology commonly used by artificial intelligence models. The correct and rigorous reference files, materials, etc. templates can be stored in the vector library that can quickly compare the similarity. When using the large language model to process tasks, the relevant materials in the vector library are retrieved by the large language model as templates or context to answer questions, avoiding the generation of incorrect answers by the large language model.
[0089] Specifically, the log format, log file template, log description, log analysis description, and log abstract template are vectorized and stored in the vector library. After the user inputs the prompt word, the large language model reads in before, and the relevant materials in the vector library are queried by the large language model, and the relevant materials are supplemented to the context of the large language model input. The prompt word with context and reference materials is sent to the large language model to generate an answer. The prompt word is a prompt instruction for the large language model to generate an abstract.
[0090] Through the above steps S210 to S240, the shard abstract of the log shard can be obtained, and the log abstract of the log file can be generated by summarizing the shard abstract of each log shard. The retrieval of the archived log is accelerated by retrieving the log abstract.
[0091] Please refer to Figure 3 In some embodiments, step S210 can include but is not limited to steps S310 to S340:
[0092] Step S310, obtaining the number of shards of the log shard;
[0093] Step S320, according to the number of shards and the preset abstract generation model, performing abstract generation on the log shard to obtain a shard abstract generation duration;
[0094] Step S330, according to the shard abstract generation duration and the number of shards, determining a target duration;
[0095] Step S340, according to the target duration and the preset expected duration, calculating the thread concurrency.
[0096] In step S310 of some embodiments, the number of log shards obtained by cutting the log file is obtained, and the number of shards is obtained, which is an integer greater than or equal to 1.
[0097] In step S320 of some embodiments, if the number of fragments is equal to 1, the log fragments are input into the summary generation model, and the summary generation model is used to generate summaries of the log fragments, to obtain a summary generation time length of the fragments. The summary generation time length is the time length consumed by the summary generation model to generate summaries of the log fragments. If the number of fragments is greater than 1, the summary generation model is used to generate summaries of each log fragment respectively, to obtain a summary generation time length of each log fragment, and the summary generation time length of each log fragment is averaged to obtain a summary generation time length of the fragments. The summary generation time length of the fragments is the average time length consumed by the summary generation model to generate summaries of one log fragment. The summary generation model is used to create a short version in a longer log fragment text while retaining the core information and main points of the original text. The summary generation model can be a BERT model, a long short-term memory network, a recurrent neural network, etc.
[0098] In step S330 of some embodiments, the summary generation time length of the fragments is multiplied by the number of fragments to obtain a target time length. The target time length is an estimated total time length consumed by the summary generation of all log fragments of the log file.
[0099] In step S340 of some embodiments, the target time length is divided by an expected time length to obtain a thread concurrency degree. The expected time length is a total time length expected to be consumed by the summary generation of all log fragments of the log file. The expected time length is less than or equal to the target time length.
[0100] Through the above steps S310 to S340, the thread concurrency degree can be obtained, so that the thread is started based on the thread concurrency degree to generate summaries of the log fragments, and the summary generation efficiency of the log fragments is improved.
[0101] Please refer to Figure 4 In some embodiments, the summary generation model includes an encoder and a decoder, and step S320 can include but is not limited to steps S410 to S440:
[0102] Step S410, filtering the log fragments according to the number of fragments to obtain selected fragments;
[0103] Step S420, feature encoding of the selected fragments by the encoder to obtain log encoding features;
[0104] Step S430, feature decoding of the log encoding features by the decoder to obtain candidate summaries of the selected fragments, and obtaining a generation time length of the candidate summaries to obtain a candidate time length;
[0105] Step S440, calculating the summary generation time length of the fragments according to the candidate time length.
[0106] In step S410 of some embodiments, if the number of shards is greater than 1, the selected shards are obtained by screening the log shards, and the shard summary generation time is calculated based on the selected shards instead of all the log shards, thereby improving the calculation efficiency. Specifically, if the number of shards is greater than 1, a first preset number of log shards are randomly selected to obtain the selected shards. The first preset number can be defined by the actual situation, such as 50% of the number of shards. The log shards can also be grouped to obtain shard groups, and a second preset number of log shards are randomly selected from each shard group to obtain the selected shards. The second preset number can be defined by the actual situation, such as 50% of the number of log shards in the shard group.
[0107] In step S420 of some embodiments, the selected shards are input to the encoder for feature encoding to capture the text semantics of the selected shards, thereby obtaining the log encoded features. The encoder can be a transformer model.
[0108] In step S430 of some embodiments, the decoder is used to understand the log encoded features and generate summaries, the log encoded features are input to the decoder for feature decoding to generate summaries, thereby obtaining candidate summaries of the selected shards, and the generation time of the candidate summaries is counted to obtain the candidate time. The decoder can use a sequence-to-sequence model, or a model opposite in structure to the encoder.
[0109] In step S440 of some embodiments, the mean value of the candidate time of each selected shard is calculated to obtain the shard summary generation time. If the candidate times of the three selected shards are x1, x2 and x3 respectively, the shard summary generation time is (x1+x2+x3) / 3, where / represents division.
[0110] Through the above steps S410 to S440, the shard summary generation time of the log shards can be quickly calculated to calculate the thread concurrency degree based on the shard summary generation time, thereby improving the efficiency of summary generation.
[0111] Please refer to Figure 5 In some embodiments, step S420 can include but is not limited to steps S510 to S530:
[0112] Step S510, the selected shards are vectorized by the encoder to obtain shard feature vectors;
[0113] Step S520, attention calculation is performed on the shard feature vectors to obtain shard attention features;
[0114] Step S530, feature extraction is performed on the shard attention features to obtain log encoded features.
[0115] In step S510 of some embodiments, the encoder comprises an embedding layer, a multi-head attention mechanism, and a feed-forward neural network. The selected shards are vectorized by the embedding layer to obtain shard feature vectors.
[0116] In step S520 of some embodiments, a relative position encoding vector of the shard feature vectors is obtained. The relative position encoding vector and the shard feature vectors are vector-aligned and added to obtain an intermediate feature vector. The multi-head attention mechanism is used to perform multi-head attention calculation on the intermediate feature vector to obtain shard attention features. Specifically, the attention calculation can be performed on the intermediate feature vector by each attention head to obtain sub-attention features, and the sub-attention features of each attention head are spliced to obtain the shard attention features.
[0117] In step S530 of some embodiments, the intermediate feature vector and the shard attention features are added to obtain a first fusion feature. The first fusion feature is batch normalized, and the normalized first fusion feature is fed into the feed-forward neural network to extract feed-forward features. The normalized first fusion feature and the feed-forward features are added to obtain a second fusion feature. The second fusion feature is normalized to obtain a log encoding feature.
[0118] Through the above steps S510 to S530, the log encoding features of the log shards can be extracted to generate a log summary based on the log encoding features.
[0119] Please refer to Figure 6 In some embodiments, step 340 can include but is not limited to steps S610 to S630:
[0120] Step S610: Calculate the initial concurrency according to the target duration and the preset expected duration;
[0121] Step S620: Obtain system load data;
[0122] Step S630: Update the initial concurrency according to the system load data to obtain the thread concurrency.
[0123] In step S610 of some embodiments, the target duration and the expected duration are divided to obtain the initial concurrency.
[0124] In step S620 of some embodiments, the system load data is obtained, which is used to evaluate the system load situation, including resource utilization and memory occupancy.
[0125] In step S630 of some embodiments, a first utilization threshold and a second utilization threshold are set for the resource utilization, the first utilization threshold is less than the second utilization threshold, and both the first utilization threshold and the second utilization threshold are greater than 0 and less than 1, the first utilization threshold can be 40%, and the second utilization threshold can be 70%. A first occupancy threshold and a second occupancy threshold are set for the memory occupancy, the first occupancy threshold is less than the second occupancy threshold, and both the first occupancy threshold and the second occupancy threshold are greater than 0 and less than 1, the first occupancy threshold can be 60%, and the second occupancy threshold can be 80%.
[0126] If the resource utilization is less than the first utilization threshold, or the memory occupancy is less than the first occupancy threshold, it indicates that the system is in a low load state, and in order to improve the utilization of system resources, the initial concurrency is gradually increased by 1 until the resource utilization is greater than or equal to the first utilization threshold, or the memory occupancy is greater than or equal to the first occupancy threshold, and then the specific processing logic is jumped to.
[0127] If the resource utilization is greater than or equal to the first utilization threshold and less than the second utilization threshold, or the memory occupancy is greater than or equal to the first occupancy threshold and less than the second occupancy threshold, it indicates that the system is in a medium load state, and the initial concurrency is taken as the thread concurrency.
[0128] If the resource utilization is greater than or equal to the second utilization threshold, or the memory occupancy is greater than or equal to the second occupancy threshold, it indicates that the system is in a high load state, and the initial concurrency is gradually decreased by 1 until the resource utilization is less than the second utilization threshold, or the memory occupancy is less than the second occupancy threshold, and then the specific processing logic is jumped to.
[0129] Through the above steps S610 to S630, the thread concurrency can be dynamically adjusted according to the system load condition.
[0130] In step S140 of some embodiments, the large language model includes a second model, and the log format, the shard summary, and a preset summary prompt instruction are input into the second model. The second model is used to generate a summary of the log format, the shard summary, and the summary prompt instruction, so as to summarize the shard summaries of the various log shards and obtain a log summary of the log file. The summary prompt instruction is used to prompt the large language model for the summary generation requirement of the log file. The summary prompt instruction can be: "analyze and summarize {context}, perform summary analysis, arrange in chronological order, and mainly highlight abnormal frequency, abnormal items, and abnormal time consumption increase", and context is the shard summary of the log shard. The log summary can be stored in a document database or other storage and retrieval system, so as to facilitate multiple reuse of the log summary.
[0131] Please refer to Figure 7In some embodiments, step S150 can include, but is not limited to, steps S710-S730:
[0132] In step S710, a file generation duration of the log file is obtained.
[0133] In step S720, an archiving parameter of the log file is determined according to the file generation duration.
[0134] In step S730, the log file is archived according to the archiving parameter to obtain an archived log.
[0135] In step S710 of some embodiments, a file generation duration of the log file is obtained, the file generation duration being a duration that the log file spans from creation to a current time.
[0136] In step S720 of some embodiments, the log usage frequency is lower and lower with time, and as time goes on, the older logs are stored on cheaper storage devices to reduce storage pressure. The present embodiment sets a multi-level log archiving strategy, determines an archiving parameter of the log file according to the file generation duration, and the archiving parameter is used to control the archiving strategy of the log file. A first duration threshold and a second duration threshold are set, the first duration threshold being smaller than the second duration threshold. If the file generation duration is smaller than the first duration threshold, the archiving parameter is determined to be local storage. If the file generation duration is greater than or equal to the first duration threshold and smaller than the second duration threshold, the archiving parameter is determined to be log system storage. If the file generation duration is greater than or equal to the second preset duration, the archiving parameter is determined to be archive storage. The first duration threshold and the second duration threshold can be set according to actual conditions, for example, the first duration threshold is 5 days and the second duration threshold is 2 weeks.
[0137] In step S730 of some embodiments, if the archiving parameter is local storage, the log file is stored in a local file system to obtain the archived log. If the archiving parameter is log system storage, the log file is stored in a first storage medium, which can be a centralized log management system built by an ELK architecture. If the archiving parameter is archive storage, the log file is stored in a second storage medium, which can be an archive storage system. The cost and query performance of the second storage medium are lower than those of the first storage medium.
[0138] The above steps S710-S730 archive the log file, reduce the occupation of storage space, and reduce the storage pressure.
[0139] In step S160 of some embodiments, as the business continues to expand, the number of log files generated by the business system continues to increase, and in order to further reduce the storage pressure, it is necessary to advance the archiving time range, such as log files generated for more than a week are compressed and archived. After advancing the compression and archiving time range, the access frequency of the archived log is higher, but the archived log after compression cannot be found, and can only be extracted and decompressed as a whole, which increases the query overhead of the log file. By generating a log summary of the log file, the archived log is searched according to the log summary, which can significantly advance the compression and archiving time range, reduce the storage pressure and storage cost of the ELK part, and improve the efficiency of log retrieval.
[0140] The embodiments of the present application extract the features of the log file in the form of a summary, so that the log can be transferred to a low-cost storage medium as soon as possible or saved in a higher compression ratio, and the summary as a log feature can accelerate the retrieval and use of the archived log. The summary is used to improve the efficiency of finding the archived log, and assists in log analysis, which can significantly reduce the archiving time range of non-local but not yet archived logs, saving a large amount of ELK storage and computing resources, and also significantly improving the efficiency of finding the archived and compressed log.
[0141] Please refer to Figure 8 The embodiments of the present application also provide a log retrieval device, which can implement the log retrieval method described above, and the log retrieval device comprises:
[0142] The acquisition module 810 is configured to acquire the log file and the log format of the log file.
[0143] The segmentation module 820 is configured to segment the log file to obtain log fragments.
[0144] The log analysis module 830 is configured to perform log analysis on the log format, the log fragments, and a preset fragment prompt instruction of the log fragments by using a pre-trained large language model to obtain a fragment summary of the log fragments.
[0145] The summary generation module 840 is configured to perform summary generation on the log format, the fragment summary, and a preset summary prompt instruction by using a large language model to obtain a log summary of the log file.
[0146] The archiving module 850 is configured to archive the log file to obtain an archived log.
[0147] The log retrieval module 860 is configured to perform log retrieval on the archived log according to the log summary.
[0148] The specific implementation of the log retrieval device is basically the same as the specific embodiments of the log retrieval method described above, and will not be repeated here.
[0149] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor executes the computer program to realize the log retrieval method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.
[0150] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:
[0151] The processor 910 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.
[0152] The memory 920 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910 to implement the log retrieval method of the embodiments of the present application.
[0153] The input / output interface 930 is used to realize information input and output.
[0154] The communication interface 940 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).
[0155] The bus 950 is used to transmit information between various components (such as the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.
[0156] The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 to realize the communication connection between them in the device.
[0157] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the log retrieval method.
[0158] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0159] The log retrieval method and device, electronic equipment and computer storage medium provided by the embodiments of the present application are used to acquire a log file and a log format of the log file, so as to understand the log file based on the log format. In order to make the log file meet the model input requirement, the log file is segmented to obtain log fragments. In order to generate an abstract that meets the business requirement, an abstract prompt instruction is constructed, and a pre-trained large language model is used to perform log analysis on the log format, the log fragments and a preset fragment prompt instruction of the log fragments, to obtain a fragment abstract of the log fragments. The large language model is used to generate an abstract based on the log format, the fragment abstract and the preset abstract prompt instruction, to summarize the abstracts of the multiple log fragments and obtain a log abstract of the log file. In order to reduce the storage cost of the log file, the log file is archived to obtain an archived log. The archived log is retrieved based on the log abstract, and the efficiency of log retrieval is improved.
[0160] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.
[0161] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps or different steps.
[0162] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.
[0163] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0164] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0165] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0166] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0167] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
[0168] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.
[0169] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.
[0170] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.
Claims
1. A log retrieval method, characterized in that, The method includes: Obtain the log file and the log format of the log file; The log file is split to obtain log fragments; Log analysis is performed on the log format, log fragments, and preset fragment prompts for the log fragments using a pre-trained large language model to obtain fragment summaries of the log fragments. The log summary of the log file is obtained by generating a summary from the log format, the fragmented summary, and the preset summary prompt instructions using the large language model. The log files are archived to obtain archived logs; Log retrieval is performed on the archived logs based on the log summary.
2. The method according to claim 1, characterized in that, The process involves analyzing the log format, log segments, and preset segmentation prompts for the log segments using a pre-trained large language model to obtain segment summaries for the log segments, including: Obtain the thread concurrency level, and start the target thread based on the thread concurrency level; The reference shard is obtained by acquiring the log shard through the target thread; The preset fragment prompt instructions are filtered according to the reference fragment to obtain the reference prompt instructions; The target thread calls the large language model, and the called large language model performs log analysis on the log format, the reference fragments, and the reference prompts to obtain the fragment summary.
3. The method according to claim 2, characterized in that, The process of obtaining thread concurrency includes: Get the number of log shards; Based on the number of shards and a preset summary generation model, a summary is generated for each log shard to obtain the shard summary generation time. The target duration is determined based on the fragment summary generation duration and the number of fragments; The thread concurrency is calculated based on the target duration and the preset expected duration.
4. The method according to claim 3, characterized in that, The digest generation model includes an encoder and a decoder. The step of generating digests for the log fragments based on the number of fragments and the preset digest generation model, to obtain the fragment digest generation time, includes: The log shards are filtered according to the number of shards to obtain the selected shards; The selected segment is feature-encoded using the encoder to obtain log coding features; The log encoding features are decoded by the decoder to obtain the candidate digest of the selected segment, and the generation time of the candidate digest is obtained to obtain the candidate duration; The fragment summary generation time is calculated based on the candidate duration.
5. The method according to claim 4, characterized in that, The step of performing feature encoding on the selected segment using the encoder to obtain log encoding features includes: The selected slice is vectorized by the encoder to obtain the slice feature vector; Attention is calculated on the segmented feature vectors to obtain segmented attention features; Feature extraction is performed on the segmented attention features to obtain the log encoding features.
6. The method according to claim 3, characterized in that, The step of calculating the thread concurrency based on the target duration and the preset expected duration includes: Calculate the initial concurrency based on the target duration and the preset expected duration; Obtain system load data; The initial concurrency is updated based on the system load data to obtain the thread concurrency.
7. The method according to any one of claims 1 to 6, characterized in that, The process of archiving the log file to obtain archived logs includes: Obtain the file generation time of the log file; The archiving parameters of the log file are determined based on the file generation time. The log file is archived according to the archiving parameters to obtain the archived log.
8. A log retrieval device, characterized in that, The device includes: The acquisition module is used to acquire the log file and the log format of the log file; The segmentation module is used to segment the log file to obtain log fragments; The log analysis module is used to perform log analysis on the log format, the log fragments, and the preset fragment prompts of the log fragments using a pre-trained large language model, so as to obtain the fragment summary of the log fragments. The summary generation module is used to generate a summary of the log file by using the large language model to analyze the log format, the fragmented summary, and the preset summary prompt instructions. The archiving module is used to archive the log files to obtain archived logs; The log retrieval module is used to perform log retrieval on the archived logs based on the log summary.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.