Log data processing method, system and equipment and medium
By using a pre-trained language model and dynamically updated prompt word templates, the problem of insufficient applicability and accuracy in log data processing is solved, achieving efficient processing and low resource consumption of log data from different sources.
Patent Information
- Application Number
- CN202510988516.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-17
- Publication Date
- 2025-11-21
AI Technical Summary
Existing technologies are not applicable or accurate enough when processing log data from different sources, are difficult to adapt to changes in log structure, and consume high computational resources.
By acquiring log data and the first prompt word template, a pre-trained language model is used for structured extraction, and the prompt word template is dynamically updated based on the structured data, thus enabling the processing of log data from different sources.
It improves the applicability and accuracy of log data processing, reduces the consumption of computing resources, and can adapt to changes in the source and context of log data.
Smart Images
Figure CN120996003A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, system, device, and medium for processing log data. Background Technology
[0002] With the continuous development of information technology, the sources of log data are becoming increasingly diverse. The log formats of log data from different sources vary greatly, making the processing of log data from different sources a key focus for system maintenance personnel.
[0003] Currently, related technologies typically employ rule-based or machine learning-based log processing methods to process log data. However, these methods are less effective at processing heterogeneous log data from different sources, and their applicability and accuracy are unsatisfactory.
[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention
[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.
[0006] Therefore, one objective of this invention is to provide a method, system, device, and medium for processing log data, wherein the method can effectively improve the applicability and accuracy of log data processing.
[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:
[0008] In a first aspect, embodiments of this application provide a method for processing log data, including:
[0009] Obtain log data and a first prompt word template, the first prompt word template being used to represent the previous second prompt word template;
[0010] The log data and the first prompt word template are input into a pre-trained language model for structured extraction to obtain the structured data output by the pre-trained language model;
[0011] Based on the log data and structured data, the first prompt word template is updated to obtain the second prompt word template;
[0012] The processing result of the log data is obtained based on the structured data.
[0013] In addition, the method according to the above embodiments of this application may also have the following additional technical features:
[0014] Furthermore, in one embodiment of this application, obtaining the first prompt word template includes:
[0015] Obtain several second prompt word templates of different template types;
[0016] Type feature extraction is performed on the log data to obtain the target template type;
[0017] Based on the target template type, all second prompt word templates are filtered to obtain the first prompt word template corresponding to the log data.
[0018] Furthermore, in one embodiment of this application, the step of inputting the log data and the first prompt word template into a pre-trained language model for structured extraction to obtain the structured data output by the pre-trained language model includes:
[0019] Based on the first prompt word template, the log data is processed to construct a prompt message, resulting in a log prompt message;
[0020] The log prompts are input into the pre-trained language model for structured extraction to obtain the structured data.
[0021] Furthermore, in one embodiment of this application, the step of updating the first prompt word template based on the log data and the structured data to obtain the second prompt word template includes:
[0022] Obtain the tag data corresponding to the structured data;
[0023] Based on the tag data and the structured data, log features are extracted from the log data to obtain tag feature data;
[0024] Based on the tag feature data, the first prompt word template is mutated and updated to obtain the second prompt word template.
[0025] Furthermore, in one embodiment of this application, the step of extracting log features from the log data based on the tag data and the structured data to obtain tag feature data;
[0026] Based on the tag data and the structured data, the log data is processed to construct a four-tuple to obtain a log tag four-tuple;
[0027] Semantic features are extracted from the log tag quadruple to obtain the tag feature data.
[0028] Furthermore, in one embodiment of this application, the step of modifying and updating the first prompt word template according to the tag feature data to obtain the second prompt word template includes:
[0029] Based on the tag feature data, a sample is constructed for the first prompt word template to obtain a tag strategy sample;
[0030] The label strategy sample is input into the policy network to perform sample prompt word mutation, and several candidate strategy samples are output by the policy network.
[0031] All the candidate strategy samples are input into a value network for sample value analysis to obtain several original sample values output by the value network, and each original sample value corresponds to one candidate strategy sample.
[0032] Based on all the original sample values and the candidate strategy samples, the first prompt word template is updated to obtain the second prompt word template.
[0033] Further, in one embodiment of this application, the step of updating the first prompt word template based on all the original sample values and the candidate strategy samples to obtain the second prompt word template includes:
[0034] The values of all the original samples are compared to obtain the target sample value, which is the largest original sample value among all the original sample values.
[0035] Based on the target sample value, all candidate strategy samples are mapped and filtered to obtain the target strategy sample;
[0036] Based on the sample prompt words in the target strategy sample, the first prompt word template is updated to obtain the second prompt word template.
[0037] Secondly, embodiments of this application provide a log data processing system, including:
[0038] The first processing unit acquires log data and a first prompt word template, wherein the first prompt word template is used to represent the previous second prompt word template;
[0039] The second processing unit is used to input the log data and the first prompt word template into a pre-trained language model for structured extraction, so as to obtain the structured data output by the pre-trained language model.
[0040] The third processing unit is used to update the first prompt word template based on the log data and structured data to obtain the second prompt word template.
[0041] The fourth processing unit is used to obtain the processing result of the log data based on the structured data.
[0042] Thirdly, embodiments of this application also provide an electronic device, including:
[0043] At least one processor;
[0044] At least one memory for storing at least one program;
[0045] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.
[0046] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.
[0047] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:
[0048] This application discloses a method, system, device, and medium for processing log data. The method involves acquiring log data and a first prompt word template, whereby the first prompt word template represents a previous second prompt word template. The log data and the first prompt word template are input into a pre-trained language model for structured extraction, resulting in structured data output by the pre-trained language model. Based on the log data and the structured data, the first prompt word template is updated to obtain a second prompt word template. Finally, the processing result of the log data is obtained based on the structured data. This method uses a pre-trained language model to perform structured extraction on log data using the latest obtained first prompt word template, enabling processing of log data from different sources and effectively improving the applicability of log data processing. Furthermore, the method dynamically updates the prompt word template using the log data and structured data, which adapts to changes in log data caused by changes in the source, effectively improving the accuracy of log data processing. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0050] Figure 1 A flowchart illustrating a log data processing method provided in an embodiment of this application;
[0051] Figure 2 A schematic diagram of the framework of a log data processing system provided in an embodiment of this application;
[0052] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0053] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0055] Currently, related technologies typically employ rule-based or machine learning-based log processing methods to process log data.
[0056] Rule-based log processing primarily relies on predefined regular expressions or Grok patterns to match logs and extract key fields. It then processes the log data using static thresholds or keyword matching. However, due to significant format differences in log data from various sources (such as Nginx, Kubernetes, and MySQL), fixed regular expressions cannot adapt to changes in log structure. This means the method cannot flexibly identify and process log data without predefined parameters, limiting its applicability. Furthermore, this method cannot identify complex attack chains (e.g., a single harmless log entry may pose a threat when combined with multiple entries) and suffers from a high false positive rate (e.g., normal business peaks are misjudged as abnormal), resulting in low accuracy in log data processing.
[0057] Machine learning-based log processing typically uses TF-IDF or Word2Vec to vectorize log text and then employs SVM, random forest, or unsupervised clustering (such as K-means clustering) to process the log data. However, this approach struggles with unstructured log data, has poor parsing capabilities for heterogeneous log data, and has limited applicability. Furthermore, it is difficult to distinguish between real threats and noise (such as temporary network jitter logs) in log data, which can easily generate a large number of invalid alarms, resulting in low accuracy in log data processing.
[0058] In addition, there are some related technologies that process log data through large language models. However, this approach usually requires a large amount of log training data to fine-tune the large language model. During the application of the large language model, it needs to be updated through continuous learning techniques. This approach requires a lot of computing resources and is costly in the process of fine-tuning and continuous learning.
[0059] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.
[0060] In view of this, embodiments of this application provide a method, system, device, and medium for processing log data. The method continuously updates the prompt word template during log data processing and uses a pre-trained language model to perform structured extraction of log data using the latest first prompt word template. This enables processing of log data from different sources, effectively improving the applicability of log data processing. Furthermore, the method, through continuously updated prompt word templates—specifically dynamically updated based on log data and structured data—can adapt to changes in log data caused by changes in source and capture the contextual relationships between multiple log data entries, effectively improving the accuracy of log data processing. Moreover, by continuously updating the prompt word template, this method eliminates the need for retraining and continuous updates to the pre-trained language model, effectively reducing the required computational resources and saving costs.
[0061] Reference Figure 1 In this application embodiment, a method for processing log data includes:
[0062] Step 110: Obtain log data and a first prompt word template, wherein the first prompt word template is used to represent the previous second prompt word template;
[0063] In this embodiment of the application, log data can be processed through a log server. Specifically, the log data can be log data from at least one source (such as Nginx, Kubernetes, MySQL), and the log data can be mixed structured data (such as JSON logs) or unstructured data (such as stack trace logs). The first prompt word template can be the most recently updated prompt word template, specifically the second prompt word template updated during the previous log data processing.
[0064] In some embodiments, obtaining the first prompt word template includes:
[0065] Obtain several second prompt word templates of different template types;
[0066] Type feature extraction is performed on the log data to obtain the target template type;
[0067] Based on the target template type, all second prompt word templates are filtered to obtain the first prompt word template corresponding to the log data.
[0068] In this embodiment of the application, when the log server receives and processes log data from different sources, it can also maintain and update several second prompt word templates of different template types. The specific template type of the second prompt word template can be template types such as error root cause analysis, security attack detection, and performance bottleneck location.
[0069] Understandably, type feature extraction can involve extracting features from log data and determining the target template type based on these features. This target template type indicates the template type corresponding to the log data. Specifically, if the log data contains features such as "error," "fail," or "fatil," the determined target template type could be an error root cause analysis template type; or, if the log data contains a frequently repeated IP feature, the determined target template type could be a security attack detection template type; or, if the response time feature in the log data is greater than 1 second, the determined target template type could be a performance bottleneck location template type.
[0070] It should be noted that template filtering can be performed by selecting the second prompt word template that corresponds to the target template type from all second prompt word templates, and then recording the second prompt word template that corresponds to the target template type as the first prompt word template.
[0071] Step 120: Input the log data and the first prompt word template into the pre-trained language model for structured extraction to obtain the structured data output by the pre-trained language model;
[0072] In this embodiment of the application, structured data from log data from any source can be extracted based on the first prompt word template through a pre-trained language model. The pre-trained language model can be a large language model (LLM). There are various types of large language models, which will not be described in detail here.
[0073] In some embodiments, the step of inputting the log data and the first prompt word template into a pre-trained language model for structured extraction to obtain structured data output by the pre-trained language model includes:
[0074] Based on the first prompt word template, the log data is processed to construct a prompt message, resulting in a log prompt message;
[0075] The log prompts are input into the pre-trained language model for structured extraction to obtain the structured data.
[0076] In this embodiment, the prompt construction process can be to add log data to a preset position in the first prompt template to obtain a log prompt, for example, the log prompt is "This is a Kubernetes container log, please determine its event type (options: OOM / crash / network failure): [2025-01-01 12:00:00] Container terminated due to memory limit"; structured extraction can be to input the log prompt into a pre-trained language model so that the pre-trained language model analyzes the log data in the log prompt based on the prompt words in the log prompt to obtain structured data output by the pre-trained language model, for example, the structured data is {"type": "OOM"; "confidence": 0.95; "action": "scale_up"}.
[0077] Step 130: Based on the log data and structured data, update the first prompt word template to obtain the second prompt word template;
[0078] In this embodiment of the application, after the received log data is extracted in a structured manner, the first prompt word template can be updated based on the determined structured data and the original log data to obtain the second prompt word template. The second prompt word template is used as the corresponding template type and is the first prompt word template for the next log data to be extracted in a structured manner.
[0079] In some embodiments, updating the first prompt word template based on the log data and the structured data to obtain the second prompt word template includes:
[0080] Obtain the tag data corresponding to the structured data;
[0081] Based on the tag data and the structured data, log features are extracted from the log data to obtain tag feature data;
[0082] Furthermore, based on the tag data and the structured data, log features are extracted from the log data to obtain tag feature data;
[0083] Based on the tag data and the structured data, the log data is processed to construct a four-tuple to obtain a log tag four-tuple;
[0084] Semantic features are extracted from the log tag quadruple to obtain the tag feature data.
[0085] In this embodiment, the label data can be real labels corresponding to structured data. This label data can be obtained by predicting and classifying structured data or log data through a machine learning model, or it can be generated and corrected manually. Log feature extraction can first involve constructing log label quadruples corresponding to log data and label data. Specifically, the log label quadruple can be [log id, misclassification, correct label, context]. The "log id" and "context" of the log label quadruple can be determined based on the log data; the "misclassification" can be determined based on the structured data; and the "correct label" can be obtained based on the label data.
[0086] It is understandable that semantic feature extraction can be performed when the "misclassification" and "correct label" in the log label quadruple are inconsistent. In this way, the semantic features of the log label quadruple can be extracted by a neural network model (such as RNN model, CNN model, Transformer model) to obtain the log label quadruple in the form of an embedded vector, which is denoted as label feature data.
[0087] It should be noted that if the "misclassification" in the log label quadruple is consistent with the "correct label", it means that the structured data output by the pre-trained language model is correctly classified. In this case, the current first prompt word template can be regarded as the second prompt word template.
[0088] Based on the tag feature data, the first prompt word template is mutated and updated to obtain the second prompt word template.
[0089] Further, the step of modifying and updating the first prompt word template based on the tag feature data to obtain the second prompt word template includes:
[0090] Based on the tag feature data, a sample is constructed for the first prompt word template to obtain a tag strategy sample;
[0091] The label strategy sample is input into the policy network to perform sample prompt word mutation, and several candidate strategy samples are output by the policy network.
[0092] All the candidate strategy samples are input into a value network for sample value analysis to obtain several original sample values output by the value network, and each original sample value corresponds to one candidate strategy sample.
[0093] In this embodiment, sample construction can involve concatenating a first prompt word template with label feature data to obtain a label policy sample. This label policy sample records several prompt words from the first prompt word template. Then, sample prompt word mutation can involve inputting the label policy sample into a policy network to generate several candidate policy samples through adversarial interaction. The policy network can be a Policy Network in reinforcement learning, and each candidate policy sample is a corresponding label policy sample after modification, replacement, addition, or deletion of several prompt words. The value network can be a Value Network in reinforcement learning. Sample value analysis involves inputting each candidate policy sample distribution into the value network, analyzing the structured extraction effect of the candidate policy samples through the value network, and quantifying it into the original sample value.
[0094] Based on all the original sample values and the candidate strategy samples, the first prompt word template is updated to obtain the second prompt word template.
[0095] Further, the step of updating the first prompt word template based on all the original sample values and the candidate strategy samples to obtain the second prompt word template includes:
[0096] The values of all the original samples are compared to obtain the target sample value, which is the largest original sample value among all the original sample values.
[0097] Based on the target sample value, all candidate strategy samples are mapped and filtered to obtain the target strategy sample;
[0098] Based on the sample prompt words in the target strategy sample, the first prompt word template is updated to obtain the second prompt word template.
[0099] In this embodiment, value comparison can be to compare the magnitude of the values of all original samples and record the original sample value with the largest value as the target sample value; mapping screening can be based on the correspondence between each original sample value and the candidate strategy sample, and determine the candidate strategy sample corresponding to the target sample value from all candidate strategy samples, and record it as the target strategy sample.
[0100] Understandably, after obtaining the target strategy sample, the prompts in the first prompt template can be updated based on each sample prompt in the target strategy sample to obtain the second prompt template. Specifically, for any sample prompt in the target strategy sample, if there is a corresponding prompt in the first prompt template, the corresponding prompt in the first prompt template can be replaced and updated based on that sample prompt; or, if there is no corresponding prompt in the first prompt template, the sample prompt can be added and updated to the first prompt template.
[0101] Step 140: Obtain the processing result of the log data based on the structured data.
[0102] In the embodiments of this application, for the processing of certain log data, after obtaining the structured data of the log data, the processing result of the log data can be obtained based on the structured data. The processing result can be the alarm analysis result, classification and storage result, etc. of the log data.
[0103] The following describes in detail, with reference to the accompanying drawings, a log data processing system according to an embodiment of this application.
[0104] Reference Figure 2 The log data processing system proposed in this application includes:
[0105] The first processing unit 101 acquires log data and a first prompt word template, wherein the first prompt word template is used to represent the previous second prompt word template;
[0106] The second processing unit 102 is used to input the log data and the first prompt word template into a pre-trained language model for structured extraction, so as to obtain the structured data output by the pre-trained language model.
[0107] The third processing unit 103 is used to update the first prompt word template according to the log data and structured data to obtain the second prompt word template;
[0108] The fourth processing unit 104 is used to obtain the processing result of the log data based on the structured data.
[0109] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0110] Reference Figure 3 This application also provides an electronic device, including:
[0111] At least one processor 201;
[0112] At least one memory 202 is used to store at least one program;
[0113] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.
[0114] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0115] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.
[0116] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0117] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.
[0119] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.
[0120] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.
[0121] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0123] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0124] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0125] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0126] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.
[0127] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.
Claims
1. A method for processing log data, characterized in that, include: Obtain log data and a first prompt word template, the first prompt word template being used to represent the previous second prompt word template; The log data and the first prompt word template are input into a pre-trained language model for structured extraction to obtain the structured data output by the pre-trained language model; Based on the log data and structured data, the first prompt word template is updated to obtain the second prompt word template; The processing result of the log data is obtained based on the structured data.
2. The method according to claim 1, characterized in that, Get the first prompt word template, including: Obtain several second prompt word templates of different template types; Type feature extraction is performed on the log data to obtain the target template type; Based on the target template type, all second prompt word templates are filtered to obtain the first prompt word template corresponding to the log data.
3. The method according to claim 1, characterized in that, The step of inputting the log data and the first prompt word template into a pre-trained language model for structured extraction, to obtain structured data output by the pre-trained language model, includes: Based on the first prompt word template, the log data is processed to construct a prompt message, resulting in a log prompt message; The log prompts are input into the pre-trained language model for structured extraction to obtain the structured data.
4. The method according to claim 1, characterized in that, The step of updating the first prompt word template based on the log data and the structured data to obtain the second prompt word template includes: Obtain the tag data corresponding to the structured data; Based on the tag data and the structured data, log features are extracted from the log data to obtain tag feature data; Based on the tag feature data, the first prompt word template is mutated and updated to obtain the second prompt word template.
5. The method according to claim 4, characterized in that, The step involves extracting log features from the log data based on the tag data and the structured data to obtain tag feature data. Based on the tag data and the structured data, the log data is processed to construct a four-tuple to obtain a log tag four-tuple; Semantic features are extracted from the log tag quadruple to obtain the tag feature data.
6. The method according to claim 4, characterized in that, The step of modifying and updating the first prompt word template based on the tag feature data to obtain the second prompt word template includes: Based on the tag feature data, a sample is constructed for the first prompt word template to obtain a tag strategy sample; The label strategy sample is input into the policy network to perform sample prompt word mutation, and several candidate strategy samples are output by the policy network. All the candidate strategy samples are input into a value network for sample value analysis to obtain several original sample values output by the value network, and each original sample value corresponds to one candidate strategy sample. Based on all the original sample values and the candidate strategy samples, the first prompt word template is updated to obtain the second prompt word template.
7. The method according to claim 6, characterized in that, The step of updating the first prompt word template based on all the original sample values and the candidate strategy samples to obtain the second prompt word template includes: The values of all the original samples are compared to obtain the target sample value, which is the largest original sample value among all the original sample values. Based on the target sample value, all candidate strategy samples are mapped and filtered to obtain the target strategy sample; Based on the sample prompt words in the target strategy sample, the first prompt word template is updated to obtain the second prompt word template.
8. A log data processing system, characterized in that, include: The first processing unit acquires log data and a first prompt word template, wherein the first prompt word template is used to represent the previous second prompt word template; The second processing unit is used to input the log data and the first prompt word template into a pre-trained language model for structured extraction, so as to obtain the structured data output by the pre-trained language model. The third processing unit is used to update the first prompt word template based on the log data and structured data to obtain the second prompt word template. The fourth processing unit is used to obtain the processing result of the log data based on the structured data.
9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-7.
10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-7.