Alarm prompt template optimization method, electronic device, medium and product

Abstract

This application discloses an alarm prompt template optimization method, electronic device, medium, and product. The method includes: acquiring a labeled data set including alarm data types and an initial prompt template corresponding to each alarm data type; inputting the labeled data set and the initial prompt template into a preset large language model for processing to obtain a first prediction result; dividing the labeled data set into a set of correctly judged cases and a set of incorrectly judged cases based on the first prediction result; modifying the initial prompt template according to the set of incorrectly judged cases to obtain candidate prompt templates; and iteratively filtering the candidate prompt templates and the initial prompt template according to the set of correctly judged cases and the set of incorrectly judged cases to obtain a final optimized prompt template set. This method can solve the problems of low accuracy, high iteration cost, and difficulty in adapting to dynamic and complex threats caused by relying on manual static design.

Description

Technical Field

[0001] This application relates to the field of network security technology, specifically to an alarm prompt template optimization method, electronic devices, readable storage media, and computer program products. Background Technology

[0002] As cybersecurity increasingly becomes the cornerstone of the digital world, automated analysis of massive alerts has become a necessity. Large language models have shown great potential in this field, but their analysis effectiveness is highly dependent on accurate input prompt templates. Existing technical solutions mostly rely on security experts manually writing and statically adjusting prompt templates. This method not only consumes a lot of manpower but also heavily relies on personal experience, making it difficult to systematically cover diverse and dynamically changing alert types such as port scanning and credential brute-force attacks. This results in weak generalization ability and low iteration efficiency of the generated prompt templates, making it impossible to quickly adapt to new attack patterns and reducing the efficiency of automated security operations. Summary of the Invention

[0003] In view of the above problems, this application provides an alarm prompt template optimization method, electronic device, readable storage medium and computer program product, which can solve the problems of low accuracy of judgment template, high iteration cost and difficulty in adapting to dynamic and complex threats caused by relying on manual static design.

[0004] Firstly, this application provides a method for optimizing alarm prompt templates, including: Obtain a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type; The labeled data set and the initial prompt template are input into a preset large language model for processing to obtain the first prediction result; Based on the first prediction result, the labeled data set is divided into a set of correctly judged data and a set of incorrectly judged data. The initial prompt template is modified based on the set of errors to obtain candidate prompt templates; Based on the set of correct judgments and the set of incorrect judgments, the candidate prompt templates and the initial prompt templates are iteratively filtered to obtain the final optimized prompt template set.

[0005] In the above technical solution, this method can get rid of the excessive reliance on manually written prompt templates. By combining large language models with automatic iterative filtering of labeled data, it can significantly improve the adaptability and generalization ability of prompt templates to various alarm types. This reduces the human cost of template design and iteration, while quickly responding to new attack patterns. In turn, it can effectively improve the accuracy and efficiency of automated security alarm analysis, so that the security operation system can respond to dynamic and complex network threats more efficiently.

[0006] In some implementations, obtaining a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type includes: Retrieve the raw alarm data set, including alarm data types; The original alarm data set is labeled to obtain a labeled data set; wherein each labeled data in the labeled data set includes at least the source network address, destination network address, source port, destination port, request content, response content, alarm data type, and label; Based on the labeled data set, the alarm data types are statistically analyzed to obtain a type set; An initial prompt template is created for each alarm data type based on the set of types. Based on the alarm data type, a corresponding initial prompt template is matched for each data item in the labeled data set.

[0007] In the above technical solution, the method can start from the original alarm data, obtain core features such as source address, port, and alarm type through standardized annotation, and at the same time, establish an initial prompt template based on the alarm type and complete accurate matching.

[0008] In some implementations, the step of inputting the labeled data set and the initial prompt template into a preset large language model for processing to obtain a first prediction result includes: Based on the alarm data type, the data items in the initial prompt template and the labeled data set are concatenated to obtain the input data set; The input data set is processed by a pre-defined large language model to predict each input data, and a first prediction result is obtained for each input data; wherein the first prediction result includes at least a first prediction conclusion and a first candidate word prediction probability list.

[0009] In the above technical solution, the method can accurately splice the initial prompt template and the labeled data based on the alarm data type to generate structured input data. Then, through the large language model, the prediction results containing the prediction conclusion and the candidate word probability list are output, which provides a quantitative and accurate judgment basis for the subsequent distinction between correct and incorrect sets, and improves the pertinence and reliability of template optimization.

[0010] In some implementations, dividing the labeled data set into a set of correctly judged results and a set of incorrectly judged results based on the first prediction result includes: Calculate the first prediction conclusion entropy value for each input data based on the first candidate word prediction probability list; Based on the entropy value of the first prediction conclusion and the labels in the input data, the correctness of the first prediction conclusion corresponding to each input data is determined, and a determination result is obtained. Based on the judgment result, the input data set is divided into a correct judgment set and an incorrect judgment set.

[0011] In the above technical solution, the method can accurately classify the input data by calculating the entropy value of the first prediction conclusion and combining it with the label to quantitatively evaluate the reliability of the prediction results of the large language model.

[0012] In some implementations, the step of determining the correctness of the first prediction conclusion corresponding to each piece of input data based on the entropy value of the first prediction conclusion and the annotation labels in the input data, and obtaining a determination result, includes: When the entropy value of the first prediction conclusion is lower than the preset entropy value threshold, and the first prediction conclusion is consistent with the label in the input data, the judgment result is determined to be correct. If the entropy value of the first prediction conclusion is not lower than the preset entropy value threshold, or if the first prediction conclusion is inconsistent with the label in the input data, the judgment result is determined to be incorrect.

[0013] In the above technical solution, the method can construct clear and quantitative rules for judging the correctness of prediction results through the dual judgment criteria of entropy threshold and label. This allows for the screening out of samples with low confidence in the prediction conclusion while accurately identifying samples whose prediction conclusion does not match the label, thereby ensuring the accuracy of the division between the correct set and the incorrect set.

[0014] In some implementations, modifying the initial prompt template based on the set of determined errors to obtain candidate prompt templates includes: Extract the complete original fields of each error data item in the error set; The complete original fields are matched with a preset error attribution rule library to obtain the attribution rule that matches each error data item; The attribution conclusion for each erroneous data item is obtained according to the attribution rules. Based on the attribution conclusions, generate modification actions corresponding to each of the erroneous data items; Based on the modification actions, the corresponding initial prompt template is modified and optimized to obtain the corrected candidate prompt template.

[0015] In the above technical solution, the method can accurately locate the root cause of the problem in the prompt template by matching the complete original fields of the judgment error set with the error attribution rules, and then generate targeted modification actions based on the attribution conclusions, thereby achieving targeted optimization of the initial prompt template, thus avoiding blind adjustments and effectively improving the optimization efficiency and adaptability of the candidate prompt template.

[0016] In some implementations, the step of iteratively filtering the candidate prompt templates and the initial prompt templates based on the correct judgment set and the incorrect judgment set to obtain the final optimized prompt template set includes: Calculate the accuracy index of each initial prompt template based on the set of correct judgments; The candidate suggestion template and the labeled data set are combined to obtain a combined data set. The concatenated data set is subjected to prediction processing by a preset large language model to obtain a second prediction result for each concatenated data in the concatenated data set; wherein, the second prediction result includes at least a second prediction conclusion and a second candidate word prediction probability list; The composite reward function value for each candidate prompt template is calculated based on the second prediction conclusion and the second candidate word prediction probability list; Based on the composite reward function value, all the candidate prompt templates are filtered to obtain the first optimized prompt template set; The first set of optimized prompt templates is filtered based on the composite reward function value, the accuracy index, and all the initial prompt templates to obtain the second set of optimized prompt templates; When the preset iteration termination condition is met in the current iteration round, the second set of optimization prompt templates obtained at the end is determined as the final set of optimization prompt templates.

[0017] In the above technical solution, this method can combine the accuracy index of the initial prompt template with the composite reward function value of the candidate prompt template to construct a multi-dimensional template screening system. Through iterative verification and multi-layer screening, it can retain the effective parts of the initial prompt template while selecting candidate prompt templates with stronger adaptability. At the same time, it controls the optimization rhythm by relying on the iteration termination condition, so as to finally obtain an optimized prompt template set with high accuracy and strong generalization ability, effectively improving the stability and reliability of automated alarm analysis.

[0018] In some implementations, calculating the accuracy index for each initial prompt template based on the set of correct determinations includes: Based on the number of samples in the labeled data set and the number of samples in the correctly judged set, calculate the overall accuracy of the initial prompt template applied to the labeled data set; The correct judgment set is classified according to the alarm data type to obtain a classification data set for each alarm data type. Based on the classification data set and the labeled data set, calculate the accuracy index for each initial prompt template; wherein the accuracy index includes at least precision and recall.

[0019] In the above technical solution, the method can construct a multi-dimensional accuracy evaluation system for the initial prompt template from two dimensions: overall accuracy and classification precision and recall. This allows for the measurement of the template's performance on the full set of labeled data, while accurately locating its adaptation performance under different alarm data types.

[0020] Secondly, this application provides an alarm prompt template optimization device, comprising: The acquisition unit is used to acquire a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type; The processing unit is used to input the labeled data set and the initial prompt template into a preset large language model for processing to obtain a first prediction result; A partitioning unit is used to divide the labeled data set into a set of correctly judged decisions and a set of incorrectly judged decisions based on the first prediction result. The modification unit is used to modify the initial prompt template according to the set of judgment errors to obtain a candidate prompt template; The filtering unit is used to iteratively filter the candidate prompt templates and the initial prompt templates based on the set of correct judgments and the set of incorrect judgments, so as to obtain the final optimized prompt template set.

[0021] In the above technical solution, the device can get rid of the excessive reliance on manually written prompt templates. By combining large language models with automatic iterative filtering of labeled data, it can significantly improve the adaptability and generalization ability of prompt templates to various alarm types. This reduces the human cost of template design and iteration, while quickly responding to new attack patterns. In turn, it can effectively improve the accuracy and efficiency of automated security alarm analysis, so that the security operation system can respond to dynamic and complex network threats more efficiently.

[0022] Thirdly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the alarm prompt template optimization method described in any one of the first aspects.

[0023] Fourthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the alarm prompt template optimization method described in any one of the first aspects.

[0024] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, performs the alarm prompt template optimization method described in any one of the first aspects.

[0025] The beneficial effects of this application are: it can accurately fix specific types of false alarms and missed alarms (such as misjudgment of internal network operation and maintenance traffic, and missed detection of high-risk attacks) through attribution-driven targeted editing, thereby significantly reducing the workload of manual review by security operations personnel; It can also automatically map model errors into structured editing actions (such as adding whitelists or adjusting thresholds) through pre-set security assessment and attribution rules, so that the optimization process has a clear goal orientation, thereby improving optimization efficiency; It can also match specific error attributions for each optimization action (such as "fixed 6 false alarms caused by missing internal network exclusion rules"), making the final generated Prompt and its optimization path traceable and auditable, thereby meeting the strong demand for decision transparency in security operations and improving the trust level of human-machine collaboration. It can also perform closed-loop optimization based entirely on fixed labeled alarm datasets, thus eliminating the need for real-time manual feedback or online A / B testing. This makes it more suitable for the model tuning stage before the security operation system goes live, reducing the threshold for implementation and operational risks. It can also construct a multi-dimensional reward function, incorporating high-risk detection capability, benign identification stability, prediction confidence and prompt simplicity into the assessment dimensions. At the same time, it introduces the attribution repair rate as a core incentive item, and guides the model to take into account various key performance aspects during the optimization and iteration process. This effectively avoids the model sacrificing core business objectives in pursuit of overall indicators and prevents problems such as increased false negative rates caused by overly conservative strategies. It also allows the attribution rule base to be continuously expanded as new attack methods emerge (such as adding DGA domain name recognition rules), thus giving the system long-term adaptability. Moreover, when new rules are added, there is no need to retrain the model; simply enabling the corresponding action in the optimization process is sufficient to quickly adapt to new scenarios. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the alarm prompt template optimization method in some embodiments of this application; Figure 2 This is a flowchart illustrating the alarm prompt template optimization method in some embodiments of this application; Figure 3 This is a schematic diagram of the alarm prompt template optimization device in some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0028] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0030] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0033] In scenarios involving automated analysis of network security alerts based on large models, a common approach is for security experts to manually write prompts (including general prompts covering various types of alerts such as port scanning and credential brute-force attacks, as well as refined prompts customized for single alert types), and then input the prompts and alert data into the large model to complete the automated analysis.

[0034] However, manually designed general prompts often fail to take into account the semantic specificity of different alarm types, which can easily lead to high false positives or false negatives. On the other hand, writing refined prompts separately is costly, has weak generalization, and cannot adapt to new attack patterns. At the same time, due to the lack of a systematic understanding of the effectiveness of prompts, the impact of subtle differences in prompt wording on model judgment is not modeled. Furthermore, manually static prompts cannot be iteratively optimized based on the judgment results.

[0035] To address the aforementioned technical issues, this application provides an alarm prompt template optimization method. This method can evaluate the output results of a large model by combining prediction entropy with data labels, and automatically map model errors to specific logical deficiencies through reinforcement learning, thereby enabling targeted automatic modification of the prompt, which reduces the pressure on security operations personnel and improves the effectiveness of security operations.

[0036] like Figure 1 As shown, some embodiments of this application provide a method for optimizing alarm prompt templates, which includes: S101. Obtain a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type; S102. Input the labeled data set and the initial prompt template into the preset large language model for processing to obtain the first prediction result; S103. Based on the first prediction result, divide the labeled data set into a set of correctly judged data and a set of incorrectly judged data. S104. Modify the initial prompt template according to the set of judgment errors to obtain candidate prompt templates; S105. Based on the set of correct judgments and the set of incorrect judgments, perform iterative filtering on the candidate prompt templates and the initial prompt templates to obtain the final optimized prompt template set.

[0037] In some embodiments, the labeled data set refers to a set of alarm data that contains alarm data types and each data item is labeled with core feature information and a judgment label.

[0038] In some embodiments, the initial prompt template refers to a pre-built basic prompt text based on the alarm data type, used to guide the large language model in analyzing the corresponding type of alarm data. The initial prompt template can be automatically generated by the large model; some alarm types include weak passwords, brute-force attacks, and unauthorized access.

[0039] In some embodiments, the first prediction result refers to the prediction conclusion and related auxiliary evaluation information output by the large language model after analyzing the alarm data based on the input labeled data set and the initial prompt template.

[0040] In some embodiments, the correct set refers to the subset of alarm data that matches the first prediction result with the label in the labeled data set and meets the preset evaluation criteria.

[0041] In some embodiments, the error set refers to the subset of alarm data where the first prediction result is inconsistent with the label in the labeled data set, or does not meet the preset evaluation criteria.

[0042] In some embodiments, a candidate prompt template refers to a prompt text to be filtered obtained by modifying and optimizing an initial prompt template based on the attribution conclusions of the error set.

[0043] In some embodiments, the optimized suggestion template set refers to the set of suggestion templates with high accuracy and strong generalization ability obtained by iteratively filtering candidate suggestion templates and initial suggestion templates in multiple rounds.

[0044] In some embodiments, the initial prompt template, the candidate prompt template, and the templates in the final optimized prompt template set all refer to the prompt template.

[0045] In the above embodiments, this method can get rid of the excessive reliance on manually written prompt templates. By combining large language models with automatic iterative filtering of labeled data, it can significantly improve the adaptability and generalization ability of prompt templates to various alarm types. This reduces the human cost of template design and iteration, while quickly responding to new attack patterns, thereby effectively improving the accuracy and efficiency of automated security alarm analysis, so that the security operation system can respond to dynamic and complex network threats more efficiently.

[0046] In some embodiments, obtaining a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type includes: Retrieve the raw alarm data set, including alarm data types; The original alarm data set is labeled to obtain a labeled data set; each labeled data in the labeled data set includes at least the source network address, destination network address, source port, destination port, request content, response content, alarm data type, and label. Based on the labeled data set, the alarm data types are statistically analyzed to obtain a type set; Create an initial prompt template for each alarm data type based on the type set; Match the corresponding initial prompt template for each data item in the labeled data set based on the alarm data type.

[0047] For example, the alarm data is labeled data containing manual tags, and the alarm type is the same for each detection. The remaining fields are common fields, some of which are shown in Table 1.

[0048] Table 1 Example of alarm data

[0049] In the above embodiments, the method can start from the original alarm data, obtain core features such as source address, port, and alarm type through standardized annotation, and at the same time, establish an initial prompt template based on the alarm type and complete accurate matching.

[0050] In some embodiments, the labeled dataset and the initial prompt template are input into a preset large language model for processing to obtain a first prediction result, including: Based on the alarm data type, the data items in the initial prompt template and the labeled data set are concatenated to obtain the input data set; The input data is predicted by a pre-defined large language model to obtain a first prediction result for each input data; the first prediction result includes at least a first prediction conclusion and a list of prediction probabilities for the first candidate words.

[0051] In the above embodiments, the method can accurately splice the initial prompt template and the labeled data based on the alarm data type to generate structured input data. Then, through the large language model, the prediction results containing the prediction conclusion and the candidate word probability list are output, which provides a quantitative and accurate judgment basis for the subsequent distinction between correct and incorrect sets, and improves the pertinence and reliability of template optimization.

[0052] In some embodiments, the labeled dataset is divided into a set of correctly judged datasets and a set of incorrectly judged datasets based on the first prediction result, including: Calculate the first prediction conclusion entropy value for each input data based on the first candidate word prediction probability list; Based on the entropy value of the first prediction conclusion and the labels in the input data, the correctness of the first prediction conclusion corresponding to each input data is judged, and the judgment result is obtained. Based on the judgment results, the input data set is divided into a set of correct judgments and a set of incorrect judgments.

[0053] In the above embodiments, the method can accurately classify the input data by calculating the entropy value of the first prediction conclusion and combining it with the label to quantitatively evaluate the reliability of the prediction results of the large language model.

[0054] In some embodiments, the correctness of the first prediction conclusion corresponding to each input data is determined based on the entropy value of the first prediction conclusion and the annotation labels in the input data, and a determination result is obtained, including: When the entropy value of the first prediction conclusion is lower than the preset entropy value threshold, and the first prediction conclusion is consistent with the label in the input data, the judgment result is determined to be correct; If the entropy value of the first prediction conclusion is not lower than the preset entropy value threshold, or if the first prediction conclusion is inconsistent with the label in the input data, the judgment result is determined to be incorrect.

[0055] In some embodiments, the method can calculate the confidence level of the output content of a large model using predictive entropy, thereby effectively identifying whether "the model is truly confident in its judgment" rather than simply looking at the output labels.

[0056] For example, this method first specifies the output format in the prompt, such as a conclusion: success / attempt / failure / false positive. When the large model outputs the result, it outputs the probability of each token (when calling the interface, the large model's response includes the probability of each token), performs regular expression extraction on the conclusion part, and calculates the entropy value. Examples are shown in Table 2.

[0057] Table 2 Examples of Entropy Calculation

[0058] The calculation formula is as follows: ; Where, p i The generation probability of each candidate word; H= (0.89log0.89+0.05log0.05+0.04log0.04+0.02log0.02)≈0.42; Because the maximum probability (0.89) is very high, the entropy value is low (close to 0 indicates complete certainty, log(4)≈1.39 is the maximum entropy for four-class classification); low entropy means high confidence. A threshold is set so that the judgment result is considered correct only when the entropy is below the threshold and the label prediction is accurate. The accuracy is obtained in each iteration, and the calculation method of Acc(a) is as follows: ; Among them, Acc (a) This indicates the accuracy rate when using prompt template a; N represents the total number of alarm data samples involved in the calculation; I is the index of the sample (from 1 to N, corresponding to the i-th sample); II ( The parentheses () are indicator functions. The function value is 1 when the condition inside the parentheses is true, and 0 when the condition is false. This represents the predicted label output by the large language model for the i-th sample when prompt template a is used; y i This represents the manually labeled real label corresponding to the i-th sample.

[0059] In the above embodiments, the method can construct clear and quantitative rules for judging the correctness of prediction results by using the dual criteria of entropy threshold and label. This allows for the screening out of samples with low confidence in the prediction conclusions while accurately identifying samples whose prediction conclusions do not match the labels, thereby ensuring the accuracy of the division between the correct and incorrect sets.

[0060] In some embodiments, the initial prompt template is modified according to the set of determined errors to obtain candidate prompt templates, including: Extract the complete original fields of each error data item in the error determination set; The complete original fields are matched with the preset error attribution rule library to obtain the attribution rule that matches each error data item; Obtain the attribution conclusion for each erroneous data item based on the attribution rules; Generate the corresponding modification action for each erroneous data item based on the attribution conclusions; Based on the modification actions, the corresponding initial prompt template is modified and optimized to obtain the corrected candidate prompt template.

[0061] For example, when the model's judgment results are inconsistent with the manual labels or have low confidence, this method can extract the original fields of the alarm (including source IP, request path, response code, authentication header, User-Agent, number of failures, domain characteristics, etc.) and match them against the rule base in sequence. If all contextual feature conditions of a rule are met, the corresponding attribution conclusion is triggered, and the editing action of the error attribution is recorded. Table 3 shows an example attribution table.

[0062] Table 3 Examples of Attribution

[0063] This method matches an attribution rule (e.g., R1: "Missing internal network whitelist") to each erroneous sample, generating a set of candidate structured actions A. t ={a1,a2,...,a M}, for each action a∈A t Generate a new Prompt, as shown in Table 4.

[0064] Table 4. Example diagram of new Prompt generation

[0065] In the above embodiments, the method can accurately locate the root cause of the problem in the prompt template by matching the complete original fields of the judgment error set with the error attribution rules, and then generate targeted modification actions based on the attribution conclusions, thereby achieving targeted optimization of the initial prompt template, thus avoiding blind adjustments and effectively improving the optimization efficiency and adaptability of candidate prompt templates.

[0066] In some embodiments, the candidate suggestion templates and the initial suggestion templates are iteratively filtered based on the set of correct decisions and the set of incorrect decisions to obtain the final optimized suggestion template set, including: Calculate the accuracy index for each initial prompt template based on the set of correct judgments; Data concatenation is performed based on the candidate suggestion template and the labeled data set to obtain the concatenated data set. The concatenated data set is processed by a pre-defined large language model to obtain a second prediction result for each concatenated data in the concatenated data set; wherein, the second prediction result includes at least a second prediction conclusion and a list of prediction probabilities for second candidate words; The composite reward function value for each candidate hint template is calculated based on the second prediction conclusion and the second candidate word prediction probability list; All candidate suggestion templates are filtered based on the composite reward function value to obtain the first optimized suggestion template set; The first set of optimized prompt templates is filtered based on the composite reward function value, accuracy index, and all initial prompt templates to obtain the second set of optimized prompt templates. When the preset iteration termination condition is met in the current iteration round, the second set of optimization prompt templates obtained at the end is determined as the final set of optimization prompt templates.

[0067] For example, this method can generate each edited prompt template P(a) for the original template and action. The system concatenates this template with each original alarm log in the fixed-label alarm dataset to form a complete inference input. A large language model is then called to analyze the data and obtain the predicted category and the corresponding key decision token probability distribution (obtained directly from the API interface). Based on the prediction results and the key decision token probabilities (see the evaluation results in the previous section), a composite reward function is calculated to select the optimal prompt for this round. The calculation method of the composite reward function is as follows: (1) Calculate the high-risk category High-risk category definition: hr = {attack failed, attack successful} (i.e., actual attack behavior), using the macro average F1 (Macro-F1) formula as follows: ; in, For the failed attack on F1, For a successful attack, use F1 The F1 score for each category (a metric that combines precision and recall, used to measure the balance of the evaluation results for that category) is as follows: ; Where 'c' represents the specific alarm category (such as 'attack failed', 'false alarm', etc.). Precision (c) This represents the precision of class c (the proportion of samples that the model classifies as class c, but which are actually of class c). Recall (c) This represents the recall rate of category c (actually the proportion of samples of category c that are correctly identified as category c by the model).

[0068] In the example above, this method prevents the model from "conservatively classifying all alerts as false alarms" in order to improve overall accuracy, thereby ensuring the ability to detect real attacks.

[0069] For example, calculate the average prediction entropy The formula is: ; This method can be further subdivided into: H avg correctThe average entropy of the correct samples; H avg error The average entropy of erroneous samples; Normalized forms are typically used in reward functions: 1 H avg (a) / H max ; Among them, H max =log∣Y∣; H max This represents the maximum possible value of the prediction entropy (i.e., the theoretical upper limit of the entropy for this classification task). |Y| represents the total number of categories in the classification task (for example, in alarm analysis, there are 4 categories: "success, attempt, failure, and false alarm", then |Y|=4).

[0070] In some embodiments, when calculating the "attribution alignment reward," traditional RL only looks at the overall metric, but the reward in this method pays special attention to whether the action has fixed the error it "promised to fix."

[0071] Definition: Set of Attribution Errors Suppose action a is triggered by an attribution rule (e.g., R1), then its corresponding target error set is: ε a ={i∈D|sample i is attributed to rule R} k The described error type} For example: ε a It includes 7 samples where "internal network IPs were mistakenly identified as attacks".

[0072] Definition: Number of repairs The actual number of repairs performed by action a is: ; in, This refers to the number of attribution error samples that were successfully corrected by the prompt template a; i∈ε a This refers to traversing the set of attribution errors ε. a Each sample i in the dataset; II ( ) refers to an indicator function (which takes 1 if the condition is met, and 0 otherwise); This provides a prompt for template a to predict the label for sample i; y i The true label for sample i.

[0073] In the example above, the method can determine how many samples that were originally incorrect and belong to that attribution category are now correctly classified.

[0074] For example, the compound reward function is shown below: ; Where |εa| is the total number of errors that the action should correct; ε=1e 5 Prevent division by zero; w1+w2+w3=1, w4≤1; Attribution Repair Rate Item w3 N fixed / |ε a | is used to ensure that RL does not "rely on luck to improve overall metrics while ignoring specific problems"; ||P(a)||tok refers to the number of tokens after Prompt P(a) is encoded by the large model tokenizer; Lref refers to the reference length (e.g., 256 or 512), used for normalization to make the magnitude of the penalty term comparable to other reward terms (e.g., F1∈[0,1]); w4 is a non-negative weight (usually small, such as 0.05~0.1), which controls the intensity of the penalty.

[0075] For example, Action A (adding to the whitelist): fixed 6 / 7 false alarms on the internal network → attribution repair rate = 85.7% → high reward; Action B (revised wording): Overall F1 score improved by 0.02, but only 1 / 7 was fixed → low attribution fix rate → rewards were suppressed.

[0076] For example, this method can select the Prompt with the highest reward as the starting point for the next round: P t+1 =argimax(r i ) Where, if r max ≤r(P t If 0.001 is added to δ (δ is a small threshold, such as 0.001), then convergence is determined and the iteration stops.

[0077] In the above embodiments, this method can combine the accuracy index of the initial prompt template with the composite reward function value of the candidate prompt template to construct a multi-dimensional template screening system. Through iterative verification and multi-layer screening, it can retain the effective parts of the initial prompt template while selecting candidate prompt templates with stronger adaptability. At the same time, it controls the optimization rhythm by relying on the iteration termination condition, so as to finally obtain an optimized prompt template set with high accuracy and strong generalization ability, effectively improving the stability and reliability of automated alarm analysis.

[0078] In some embodiments, the accuracy metric for each initial prompt template is calculated based on the set of correctly determined prompts, including: Based on the number of samples in the labeled dataset and the number of samples in the correctly identified dataset, calculate the overall accuracy of the initial prompt template applied to the labeled dataset; The correct alarm data set is classified according to the alarm data type to obtain the classification data set for each alarm data type. Based on the categorized and labeled datasets, calculate the accuracy metrics for each initial prompt template; the accuracy metrics include at least precision and recall.

[0079] In the above embodiments, the method can construct a multi-dimensional accuracy evaluation system for the initial prompt template from two dimensions: overall accuracy and categorical precision and recall. This allows for the measurement of the template's performance on the full set of labeled data while accurately locating its adaptation performance under different alarm data types.

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below. In some embodiments, such as Figure 2 As shown, the optimization method for this alarm prompt template includes: S201. Establish initialization prompt templates for different alarm types; S202. Analyze the labeled data using a large model; S203. Evaluate the results of the analysis; S204. Modify the prompt template until the required precision or maximum number of iterations is met.

[0081] In some embodiments, the alarm prompt template optimization method is the automatic iterative optimization method for the prompt template.

[0082] Figure 3 A schematic diagram of an alarm prompt template optimization device is shown. It should be understood that this device is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0083] The alarm prompt template optimization device includes: The acquisition unit 310 is used to acquire a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type; Processing unit 320 is used to input the labeled data set and the initial prompt template into a preset large language model for processing to obtain the first prediction result; The partitioning unit 330 is used to divide the labeled data set into a set of correctly judged decisions and a set of incorrectly judged decisions based on the first prediction result. Modification unit 340 is used to modify the initial prompt template according to the set of judgment errors to obtain candidate prompt templates; The filtering unit 350 is used to iteratively filter the candidate prompt templates and the initial prompt templates based on the set of correct judgments and the set of incorrect judgments, so as to obtain the final optimized prompt template set.

[0084] In some embodiments, the acquisition unit 310 includes: The first acquisition subunit 311 is used to acquire a set of raw alarm data including alarm data types; The annotation subunit 312 is used to annotate the original alarm data set to obtain an annotated data set; wherein each annotated data in the annotated data set includes at least the source network address, destination network address, source port, destination port, request content, response content, alarm data type, and annotation label; The statistics subunit 313 is used to statistically analyze the alarm data types based on the labeled data set to obtain a type set; Create subunit 314 to create an initial prompt template corresponding to each alarm data type based on the type set; The first matching subunit 315 is used to match the corresponding initial prompt template for each data item in the labeled data set according to the alarm data type.

[0085] In some embodiments, the processing unit 320 includes: The first splicing subunit 321 is used to splice data items in the initial prompt template and the labeled data set according to the alarm data type to obtain the input data set. The first prediction subunit 322 is used to perform prediction processing on each input data in the input data set through a preset large language model to obtain a first prediction result for each input data; wherein, the first prediction result includes at least a first prediction conclusion and a first candidate word prediction probability list.

[0086] In some embodiments, the partitioning unit 330 includes: The first calculation subunit 331 is used to calculate the first prediction conclusion entropy value of each input data according to the first candidate word prediction probability list; The judgment subunit 332 is used to judge the correctness of the first prediction conclusion corresponding to each input data based on the entropy value of the first prediction conclusion and the label in the input data, and obtain the judgment result. Sub-unit 333 is used to divide the input data set into a set of correct judgments and a set of incorrect judgments based on the judgment result.

[0087] In some embodiments, the determination subunit 332 is specifically used to determine that the determination result is correct when the entropy value of the first prediction conclusion is lower than a preset entropy value threshold and the first prediction conclusion is consistent with the label in the input data. The judgment subunit 332 is also used to determine that the judgment result is incorrect when the entropy value of the first prediction conclusion is not lower than the preset entropy value threshold, or when the first prediction conclusion is inconsistent with the label in the input data.

[0088] In some embodiments, the modification unit 340 includes: Extraction subunit 341 is used to extract the complete original fields of each error data item in the error judgment set; The second matching subunit 342 is used to match the complete original field with the preset error attribution rule library to obtain the attribution rule that matches each error data item; The second acquisition subunit 343 is used to acquire the attribution conclusion for each erroneous data item according to the attribution rules; Generating subunit 344 is used to generate the modification action corresponding to each erroneous data item based on the attribution conclusion; Modify subunit 345 to modify and optimize the corresponding initial prompt template according to the modification action, and obtain the corrected candidate prompt template.

[0089] In some embodiments, the filtering unit 350 includes: The second calculation subunit 351 is used to calculate the accuracy index of each initial prompt template based on the set of correct judgments; The second splicing subunit 352 performs data splicing processing based on the candidate prompt template and the labeled data set to obtain the spliced ​​data set. The second prediction subunit 353 is used to perform prediction processing on the concatenated data set through a preset large language model to obtain a second prediction result for each concatenated data in the concatenated data set; wherein, the second prediction result includes at least a second prediction conclusion and a second candidate word prediction probability list. The second calculation subunit 351 is also used to calculate the composite reward function value of each candidate prompt template based on the second prediction conclusion and the second candidate word prediction probability list; The filtering subunit 354 is used to filter all candidate suggestion templates based on the composite reward function value to obtain the first optimized suggestion template set; The filtering subunit 354 is also used to filter the first set of optimized prompt templates based on the composite reward function value, accuracy index, and all initial prompt templates to obtain the second set of optimized prompt templates. Subunit 355 is defined to determine the final set of second optimization prompt templates as the final set of optimization prompt templates when the preset iteration termination condition is met in the current iteration round.

[0090] In some embodiments, the second calculation subunit 351 is specifically used to calculate the overall accuracy of the initial prompt template applied to the labeled data set based on the number of samples in the labeled data set and the number of samples in the correctly determined set. The second calculation subunit 351 is further used to classify the correctly judged set according to the alarm data type to obtain the classification data set of each alarm data type. The second calculation subunit 351 is further used to calculate the accuracy index of each initial prompt template based on the classification data set and the labeled data set; wherein the accuracy index includes at least precision and recall.

[0091] like Figure 4 As shown, this application provides an electronic device 400, which includes a processor 401 and a memory 402. The processor 401 and the memory 402 are interconnected and communicate with each other through a communication bus 403 and / or other forms of connection mechanism (not shown). The memory 402 stores a computer program that can be executed by the processor 401. When the computing device is running, the processor 401 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0092] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0093] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0094] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for optimizing alarm prompt templates, characterized in that, include: Obtain a set of labeled data including alarm data types and an initial prompt template corresponding to each alarm data type; The labeled data set and the initial prompt template are input into a preset large language model for processing to obtain the first prediction result; Based on the first prediction result, the labeled data set is divided into a set of correctly judged data and a set of incorrectly judged data. Extract the complete original fields of each error data item in the error set; The complete original fields are matched with a preset error attribution rule library to obtain the attribution rule that matches each error data item; The attribution conclusion for each erroneous data item is obtained according to the attribution rules. Based on the attribution conclusions, generate modification actions corresponding to each of the erroneous data items; Based on the modification actions, the corresponding initial prompt template is modified and optimized to obtain the corrected candidate prompt template; Based on the set of correct judgments and the set of incorrect judgments, the candidate prompt templates and the initial prompt templates are iteratively filtered to obtain the final optimized prompt template set.

2. The alarm prompt template optimization method according to claim 1, characterized in that, The acquisition of the labeled data set of alarm data types and the initial prompt template corresponding to each alarm data type includes: Retrieve the raw alarm data set, including alarm data types; The original alarm data set is labeled to obtain a labeled data set; wherein each labeled data in the labeled data set includes at least the source network address, destination network address, source port, destination port, request content, response content, alarm data type, and label; Based on the labeled data set, the alarm data types are statistically analyzed to obtain a type set; An initial prompt template is created for each alarm data type based on the set of types. Based on the alarm data type, a corresponding initial prompt template is matched for each data item in the labeled data set.

3. The alarm prompt template optimization method according to claim 1, characterized in that, The step of inputting the labeled data set and the initial prompt template into a preset large language model for processing to obtain a first prediction result includes: Based on the alarm data type, the data items in the initial prompt template and the labeled data set are concatenated to obtain the input data set; The input data set is processed by a pre-defined large language model to predict each input data, and a first prediction result is obtained for each input data; wherein the first prediction result includes at least a first prediction conclusion and a first candidate word prediction probability list.

4. The alarm prompt template optimization method according to claim 3, characterized in that, The step of dividing the labeled data set into a set of correctly judged results and a set of incorrectly judged results based on the first prediction result includes: Calculate the first prediction conclusion entropy value for each input data based on the first candidate word prediction probability list; Based on the entropy value of the first prediction conclusion and the labels in the input data, the correctness of the first prediction conclusion corresponding to each input data is determined, and a determination result is obtained. Based on the judgment result, the input data set is divided into a correct judgment set and an incorrect judgment set.

5. The alarm prompt template optimization method according to claim 4, characterized in that, The step of determining the correctness of the first prediction conclusion corresponding to each piece of input data based on the entropy value of the first prediction conclusion and the labels in the input data, and obtaining a determination result, includes: When the entropy value of the first prediction conclusion is lower than the preset entropy value threshold, and the first prediction conclusion is consistent with the label in the input data, the judgment result is determined to be correct. If the entropy value of the first prediction conclusion is not lower than the preset entropy value threshold, or if the first prediction conclusion is inconsistent with the label in the input data, the judgment result is determined to be incorrect.

6. The alarm prompt template optimization method according to claim 1, characterized in that, The step of iteratively filtering the candidate prompt templates and the initial prompt templates based on the correct judgment set and the incorrect judgment set to obtain the final optimized prompt template set includes: Calculate the accuracy index of each initial prompt template based on the set of correct judgments; The candidate suggestion template and the labeled data set are combined to obtain a combined data set. The concatenated data set is subjected to prediction processing by a preset large language model to obtain a second prediction result for each concatenated data in the concatenated data set; wherein, the second prediction result includes at least a second prediction conclusion and a second candidate word prediction probability list; The composite reward function value for each candidate prompt template is calculated based on the second prediction conclusion and the second candidate word prediction probability list; Based on the composite reward function value, all the candidate prompt templates are filtered to obtain the first optimized prompt template set; The first set of optimized prompt templates is filtered based on the composite reward function value, the accuracy index, and all the initial prompt templates to obtain the second set of optimized prompt templates; When the preset iteration termination condition is met in the current iteration round, the second set of optimization prompt templates obtained at the end is determined as the final set of optimization prompt templates.

7. The alarm prompt template optimization method according to claim 6, characterized in that, The step of calculating the accuracy index for each initial prompt template based on the correct set includes: Based on the number of samples in the labeled data set and the number of samples in the correctly judged set, calculate the overall accuracy of the initial prompt template applied to the labeled data set; The correct judgment set is classified according to the alarm data type to obtain a classification data set for each alarm data type. Based on the classification data set and the labeled data set, calculate the accuracy index for each initial prompt template; wherein the accuracy index includes at least precision and recall.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the alarm prompt template optimization method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the alarm prompt template optimization method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program, which, when executed by a processor, performs the alarm prompt template optimization method according to any one of claims 1 to 7.

Images