Multi-model recommendation method and device for security alarm research and judgment, and electronic equipment

Abstract

This invention discloses a multi-model recommendation method, apparatus, and electronic device for security alarm assessment, relating to the field of artificial intelligence technology. The method includes: acquiring a baseline test set, a reference model set, and their pre-stored scores in historical versions; scheduling multiple test models with different architectures to perform assessment and inference; determining the evaluation condition equivalence index based on the rate-limited retry rate, backoff waiting time, and serialized queuing time of each test model to obtain the normalized inference latency; determining the expected operational cost based on the three-level classification prediction results, confusion probability distribution, and asymmetric misclassification cost matrix; downgrading or eliminating test cases with information fidelity scores below a first threshold; determining the equivalent alignment score of the test model on the historical version test set using the anchored use case set; using the normalized inference latency as the latency evaluation value when the evaluation condition equivalence index is below a second threshold; and determining the recommended model based on the expected operational cost, the latency evaluation value, and the equivalent alignment score.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and network security alarm analysis technology, and in particular to a multi-model recommendation method, device and electronic device for security alarm judgment. Background Technology

[0002] In security alert analysis scenarios, various large language models are available. Currently, the selection of a large language model for this scenario is typically based on the general benchmark test results published by the model provider, the subjective experience of the security team, or manual testing on a small sample. However, the actual performance of the large language model selected using these methods deviates from the recommended evaluation results. Recommendations based on subjective experience are difficult to reproduce across different personnel and timeframes, and completing a round of recommendations based on a small sample through manual testing is time-consuming.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problem that when performing security assessment and evaluation on multiple heterogeneous large language models and making recommendations accordingly, the recommended models deviate from the model with the best performance in the assessment. This invention provides a multi-model recommendation method, device and electronic device for security alarm assessment.

[0005] The first aspect of this invention provides a multi-model recommendation method for security alarm assessment, comprising: acquiring a pre-constructed baseline test set, a preset reference model set, and pre-stored scores of the reference model set on historical versions of the baseline test set; wherein the baseline test set includes multiple test cases, each test case having a three-level classification security label and an anchor test case identifier, the heterogeneous format files in the test cases being structured text representations, and each test case carrying a pre-calculated information fidelity score; scheduling multiple test models with different architectures to perform assessment and inference on the baseline test set; determining an evaluation condition equivalence index based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each test model; and subtracting the backoff waiting time and the serialization queuing time from the overall response delay of each test model to obtain a normalized inference delay; extracting three-level classification prediction results from the assessment and inference responses of each test model, and determining the expected operational cost based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security labels, and a preset asymmetric misclassification cost matrix; wherein, in the asymmetric misclassification cost matrix, The costs corresponding to at least two types of misclassification are not equal; when determining the expected operational cost, test cases with information fidelity scores lower than a preset first threshold are downweighted or removed; based on the average score of the reference model set on the anchor example set of the current version of the baseline test set, the pre-stored score, and the score of the model under test on the anchor example set of the current version, the equivalent alignment score of the model under test on the historical version test set is determined; wherein, the anchor example set is the set of test cases in the baseline test set whose anchor example is marked as true; the latency evaluation value of each model under test is determined, wherein, when the evaluation condition equivalence index is not lower than a preset second threshold, the overall response latency is used as the latency evaluation value, and when the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference latency is used as the latency evaluation value; based on the expected operational cost, the latency evaluation value, and the equivalent alignment score, a recommended model is determined from the multiple models under test.

[0006] In one embodiment of the present invention, the step of scheduling multiple test models with different architectures to perform judgment and reasoning on the baseline test set includes: for any test model among the multiple test models, converting preset evaluation prompt words into input prompt words adapted to the interface format of the test model according to the input template corresponding to the test model, and inputting them into the test model; and parsing the judgment and reasoning response output by the test model into a conclusion field and a confidence field in a preset format, wherein the conclusion field includes the three-level classification prediction result.

[0007] In one embodiment of the present invention, before obtaining the pre-constructed baseline test set, the preset reference model set, and the pre-stored scores of the reference model set on historical versions of the baseline test set, the method further includes: performing normalized text processing on the heterogeneous format file to obtain the structured text representation; determining the coverage rate of the structured text representation to the preset key field set, and the semantic retention rate of the structured text representation to the content corresponding to the key field set in the heterogeneous format file, wherein the preset key field set is a set of fields pre-defined according to the type of the heterogeneous format file and used as the basis for judgment; and determining the information fidelity score based on the coverage rate and the semantic retention rate.

[0008] In one embodiment of the present invention, both the three-level classification security label and the three-level classification prediction result include a malicious category, a suspicious category, and a safe category; wherein, in the asymmetric misclassification cost matrix, the cost corresponding to the misclassification that the true category is malicious but is predicted as safe is greater than the cost corresponding to the misclassification that the true category is malicious but is predicted as suspicious; the cost corresponding to the misclassification that the true category is safe but is predicted as malicious is greater than the cost corresponding to the misclassification that the true category is safe but is predicted as suspicious.

[0009] In one embodiment of the present invention, determining the evaluation condition equivalence index based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each model under test includes: determining a weighted sum based on the standard deviation of the rate-limited retry rate, the standard deviation of the backoff waiting time, and the standard deviation of the serialization queuing time; and determining the evaluation condition equivalence index based on the normalized weighted sum, wherein the evaluation condition equivalence index decreases as the normalized weighted sum increases.

[0010] In one embodiment of the present invention, the test cases are further labeled with a list of key evidence; wherein, determining the recommended model from the plurality of models to be tested based on the expected operating cost, the delay evaluation value, and the equivalent alignment score includes: extracting a chain of inference evidence containing evidence type, evidence value, and relevance from the judgment and reasoning response; weighting and summing the evidence items in the chain of inference evidence whose evidence type and evidence value match the list of key evidence according to the relevance to obtain the evidence chain matching degree; and determining the recommended model from the plurality of models to be tested based on the expected operating cost, the delay evaluation value, the equivalent alignment score, and the evidence chain matching degree.

[0011] In one embodiment of the present invention, the anchored test case identifier is set according to a difficulty stability index, which is determined based on the ratio of the standard deviation to the mean of the reasoning scores of the reference model set on the corresponding test case; wherein, the anchored test case identifier of the test case whose difficulty stability index is higher than a set threshold is set to true.

[0012] A second aspect of the present invention provides a multi-model recommendation device for security alarm assessment, comprising: an acquisition module for acquiring a pre-constructed baseline test set, a preset reference model set, and pre-stored scores of the reference model set on historical versions of the baseline test set; wherein the baseline test set includes multiple test cases, each test case having a three-level classification security label and an anchor test case identifier, the heterogeneous format files in the test cases being structured text representations, and each test case carrying a pre-calculated information fidelity score; a scheduling module for scheduling multiple test models with different architectures to perform assessment and inference on the baseline test set; a first determination module for determining an evaluation condition equivalence index based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each test model, and subtracting the backoff waiting time and the serialization queuing time from the overall response delay of each test model to obtain a normalized inference delay; and a second determination module for extracting three-level classification prediction results from the assessment and inference responses of each test model, and based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security labels, and a preset asymmetric misclassification. A class cost matrix determines the expected operational cost, wherein at least two types of misclassification costs in the asymmetric misclassification cost matrix are not equal, and test cases with information fidelity scores lower than a preset first threshold are downweighted or removed when determining the expected operational cost; an alignment module is used to determine the equivalent alignment score of the model under test on the historical version test set based on the average score of the reference model set on the anchoring example set of the current version of the baseline test set, the pre-stored score, and the score of the model under test on the anchoring example set of the current version, wherein the anchoring example set is the set of test cases in the baseline test set where the anchoring example is marked as true; an evaluation module is used to determine the latency evaluation value of each model under test, wherein when the evaluation condition equivalence index is not lower than a preset second threshold, the overall response latency is used as the latency evaluation value, and when the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference latency is used as the latency evaluation value; a recommendation module is used to determine a recommended model from the plurality of models under test based on the expected operational cost, the latency evaluation value, and the equivalent alignment score.

[0013] A third aspect of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-model recommendation method for security alarm assessment as described in the first aspect.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-model recommendation method for security alarm assessment as described in the first aspect.

[0015] Compared with the prior art, the technical effects achieved by the present invention are as follows:

[0016] When determining the expected operational cost, test cases with information fidelity scores below the first threshold are downweighted or eliminated, thus suppressing their contribution to the expected operational cost. The expected operational cost is further derived based on the three-level classification prediction results, confusion probability distribution, and asymmetric misclassification cost matrix. With the matrix ensuring that at least two types of misclassification costs are unequal, the evaluation results reflect the different costs corresponding to different misclassification consequences. The evaluation condition equivalence index is determined based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each model under test. When it falls below the second threshold, the latency evaluation value for recommendation is replaced by the normalized inference latency after deducting the backoff waiting time and serialization queuing time. Therefore, the latency comparison of each model under inconsistent calling conditions is unaffected by backoff and queuing. Furthermore, using an anchored example set, the score of the model under test on the current version test set is aligned to its equivalent alignment score on historical version test sets, thus providing a common comparison benchmark for evaluation results across different version test sets. The final recommendation model is determined by the aforementioned expected operational cost, latency evaluation value, and equivalent alignment score. This solves the technical problem that when performing security assessments and evaluations on multiple heterogeneous large language models and making recommendations accordingly, the recommended models deviate from the model with the best performance in the assessment. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multi-model recommendation method for security alarm analysis according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a multi-model recommendation device for security alarm analysis according to an embodiment of the present invention; Figure 3 This is a block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0018] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0019] The technical solution of the present invention is illustrated below through specific embodiments. It should be understood that the one or more steps mentioned in the present invention do not preclude the existence of other methods and steps before or after the combined steps, or that other methods and steps may be inserted between these explicitly mentioned steps. It should also be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Unless otherwise stated, the numbering of each method step is only for the purpose of identifying each method step, and not for limiting the order of each method or limiting the scope of the present invention. Changes or adjustments to their relative relationships, without substantial changes to the technical content, can also be considered as within the scope of the present invention.

[0020] In one embodiment of the present invention, a multi-model recommendation method for security alarm assessment is provided, which is used to determine a recommended model suitable for security alarm assessment among multiple models to be tested. This method can be executed by an evaluation device deployed on a cloud server.

[0021] The models under test in this embodiment are large language models with different architectures, such as models that provide services based on cloud application programming interfaces or models deployed in local inference environments. Different models under test may differ in interface form, file input capability and billing method.

[0022] like Figure 1 As shown, the method may specifically include the following steps: S101, obtain the pre-built baseline test set, the preset reference model set, and the pre-stored scores of the reference model set on the historical versions of the baseline test set.

[0023] In this embodiment, the baseline test set is a set of test cases constructed for security alarm analysis. Each test case corresponds to a security alarm to be analyzed, such as alarms in scenarios like phishing emails, web shells, abnormal logins, or suspicious host commands. The reference model set consists of several pre-selected models used to provide a reference for aligning scores across versions. The pre-stored scores are the evaluation scores of the reference model set recorded on earlier versions of the baseline test set, i.e., historical version test sets.

[0024] Each test case carries a pre-labeled three-level security category tag, derived from the categories of malicious, suspicious, and safe, indicating that the alert requires action, requires manual review, and can be closed, respectively. Optionally, each test case also carries an anchor test case identifier. Test cases with the anchor test case identifier set to true constitute the anchor test case set. This anchor test case set is retained across different versions of the baseline test set as common test cases for cross-version comparisons.

[0025] Test cases may include heterogeneous format files, such as packet capture format pcap traffic files, EML email files, or PDF documents. These heterogeneous format files are carried in the test cases in a structured text representation after being processed by standardized textualization. Each test case also carries an information fidelity score determined by the coverage and semantic retention rate of the preset key field set based on the structured text representation. This information fidelity score is a value between zero and one, used to characterize the degree of information retention after the heterogeneous format file is textualized; the closer the value is to one, the higher the degree of retention.

[0026] In addition, in this embodiment, before performing model recommendation, the annotation of each test case in the baseline test set is also subjected to quality control.

[0027] During the construction of the baseline test set, the three-level classification security labels for each test case are independently labeled by at least two labelers. Test cases with inconsistent labeling results are arbitrated by a third labeler. The labeling consistency among the labelers is calculated as a quality indicator of the baseline test set. Optionally, the list of key evidence and threat severity levels for the test cases are determined through the same independent labeling and arbitration process.

[0028] S102 schedules multiple test models with different architectures to perform analytical reasoning on the baseline test set.

[0029] In this embodiment, test cases from the baseline test set are distributed to each model under test. Each model under test then analyzes the security alarms corresponding to the test cases and records the analysis and reasoning response, analysis time, and token consumption of each model under test for each test case.

[0030] Optionally, for parallel evaluation, when the call trigger rate of a certain model under test is limited, the call to that model under test can be transferred to serial execution and backoff retry can be used, without affecting the parallel calls of other models under test.

[0031] S103, based on the rate-limited retry rate, backoff waiting time and serialization queuing time of each model under test, determine the evaluation condition equivalence index; and subtract the backoff waiting time and serialization queuing time from the overall response delay of each model under test to obtain the normalized inference delay.

[0032] Among them, the rate-limited retry rate is the proportion of subtasks retried by the model under test due to triggering the rate limit; the backoff waiting time is the duration of waiting before retrying; and the serialization queuing time is the duration of waiting in the queue after the model under test switches to serial execution. The evaluation condition equivalence index is used to characterize the degree of difference in evaluation conditions experienced by each model under test during the call process. The greater the difference in the above durations and retry rates among the models under test, the lower the evaluation condition equivalence index.

[0033] In this embodiment, the overall response latency is the time from when the request is sent to when the response is fully returned, which includes the time spent on backoff waiting and queuing; the normalized inference latency is the time obtained by deducting this part of the time affected by the calling conditions from the overall response latency, so as to reflect the inference time of the model under test itself.

[0034] S104. Extract the three-level classification prediction results from the judgment and reasoning responses of each model to be tested, and determine the expected operating cost based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security labels, and the preset asymmetric misclassification cost matrix.

[0035] In this embodiment, the three-level classification prediction result is the judgment conclusion of the model under test on the alarm corresponding to the test case as malicious, suspicious or safe; the confusion probability distribution reflects the probability that the model under test will predict it as a certain category when the true category is a certain category; the asymmetric misclassification cost matrix is ​​a matrix that configures cost values ​​for each misclassification case, wherein the costs corresponding to at least two misclassifications are not equal. For example, the cost corresponding to misclassifying an alarm that is actually malicious as safe may be different from the cost corresponding to misclassifying an alarm that is actually safe as malicious.

[0036] For example, the predicted probability under each misclassification scenario, the prior probability of the true category, and the cost corresponding to that misclassification scenario can be multiplied together and summed over all misclassification scenarios to obtain the expected operating cost. When determining the expected operating cost, test cases with information fidelity scores below a preset first threshold are downweighted or eliminated, i.e., their contribution weight to the expected operating cost is reduced, or they are not included in the calculation of the expected operating cost. Optionally, the first threshold is preset to 0.85, and test cases with an information fidelity score lower than 0.85 are considered to have insufficient information retention after textification.

[0037] S105, based on the average score and pre-stored score of the reference model set on the anchoring example set of the current version of the baseline test set, and the score of the model under test on the anchoring example set of the current version, determine the equivalent alignment score of the model under test on the historical version test set.

[0038] Because the anchoring example set remains consistent between historical and current versions, it can serve as a common benchmark connecting the two versions. For example, the equivalent alignment score of the model under test on the historical version test set is obtained by adding the score of the model under test on the current version anchoring example set to the difference between the pre-stored scores of the reference model set on the historical version test set and the average score of the reference model set on the current version anchoring example set. This allows the evaluation results obtained on the current version anchoring example set to be compared with the evaluation results on the historical version under the same benchmark.

[0039] S106, determine the delay evaluation value for each model to be tested. When the evaluation condition equivalence index is not lower than the preset second threshold, the overall response delay is used as the delay evaluation value. When the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference delay is used as the delay evaluation value.

[0040] When the evaluation condition equivalence index is not lower than the preset second threshold, the calling conditions experienced by each model under test are relatively similar, and their overall response delays are comparable. Therefore, the overall response delay is directly used as the delay evaluation value. When the evaluation condition equivalence index is lower than the preset second threshold, the calling conditions experienced by each model under test differ significantly. The portion of the overall response delay affected by backoff and queuing is not suitable for direct comparison. Therefore, the normalized inference delay is used as the delay evaluation value. Optionally, the preset second threshold is set to 0.85.

[0041] S107. Based on expected operating costs, delayed evaluation values, and equivalent alignment scores, a recommendation model is determined from multiple models to be tested.

[0042] Specifically, the evaluation equipment compares the expected operating costs, delay evaluation values, and equivalent alignment scores of each model under test, and determines the recommended model accordingly.

[0043] Optionally, the expected operating cost, delay evaluation value, and equivalent alignment score can be normalized and then weighted and summed to obtain a comprehensive evaluation of each model under test, and the model with the best comprehensive evaluation can be determined as the recommended model. In one example, the model with the best equivalent alignment score can also be selected as the recommended model from among the models under test that meet the preset upper limit of expected operating cost or upper limit of delay.

[0044] In this embodiment of the invention, test cases with information fidelity scores below a first threshold are downweighted or eliminated when determining the expected operating cost, thereby suppressing the contribution of such test cases to the expected operating cost. The expected operating cost is further derived based on the three-level classification prediction results, the confusion probability distribution, and the asymmetric misclassification cost matrix. With the matrix setting that at least two types of misclassification costs are unequal, the evaluation results can reflect the different costs corresponding to different misclassification consequences. In this embodiment, the evaluation condition equivalence index is determined based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each model under test. When it is below a second threshold, the latency evaluation value participating in the recommendation is replaced by the normalized inference latency after deducting the backoff waiting time and serialization queuing time. Therefore, the latency comparison of each model under test under inconsistent calling conditions is not affected by backoff and queuing. Furthermore, by utilizing an anchored example set, the score of the model under test on the current version test set is aligned to its equivalent alignment score on historical version test sets, thus providing a common benchmark for evaluation results across different version test sets. The final recommended model is determined jointly by the aforementioned expected operational cost, latency evaluation value, and equivalent alignment score. This addresses the technical problem of the recommended model deviating from the optimal model when performing security assessments and evaluations on multiple heterogeneous large language models and making recommendations accordingly.

[0045] In one embodiment of the present invention, multiple test models with different architectures are scheduled to perform analytical reasoning on a baseline test set, including but not limited to the following steps: S201, for any one of the multiple test models, according to the input template corresponding to the test model, convert the preset evaluation prompt words into input prompt words that are compatible with the interface format of the test model, and input them into the test model; Specifically, the preset evaluation prompts are pre-written prompts shared by all models under test. They contain the alarm content to be analyzed and the analysis instructions requiring the model under test to output an analysis conclusion, which is either malicious, suspicious, or safe. Different models under test have different interface formats; some receive input using a tag structure, some using a title level or key-value pairs, and some only receive plain text. Therefore, input templates are pre-configured for each model under test. The input template specifies the organizational structure of the input prompts for the corresponding model under test, and includes slots corresponding to the fields of the alarm content and analysis instructions in the evaluation prompts. The evaluation device fills the fields of the evaluation prompts into the corresponding slots according to the input template, obtaining input prompts that conform to the interface format of the model under test. This conversion only changes the organizational form of the prompts, without changing the alarm content and analysis instructions they carry.

[0046] For example, for a test model whose input is organized in a tag structure, its input template includes alarm content and judgment instructions into the corresponding tag paragraphs respectively; for a test model whose input is organized in a title level, its input template maps the above fields to the corresponding titles and their body text; for a test model whose input is organized in a key-value pair or structured pattern, its input template fills the above fields into the corresponding key or pattern fields.

[0047] Optionally, the above input template is pre-written manually based on the interface document of the model under test, or automatically generated according to the input format conventions in the interface document; the consistency between the converted input prompts and the preset evaluation prompts in terms of alarm content can be verified by manual spot checks.

[0048] S202, the judgment and reasoning response output by the model to be tested is parsed into a conclusion field and a confidence field in a preset format, wherein the conclusion field includes the three-level classification prediction results.

[0049] The original responses of the models under test to the input prompts vary in their expression. The evaluation device parses the original responses into a unified conclusion field and a confidence field according to a preset format: the conclusion field is taken from the judgment conclusion given by the model under test, and the value is malicious, suspicious or safe, and serves as the three-level classification prediction result of the model under test for the corresponding test case; the confidence field represents the degree of confidence of the model under test in its judgment conclusion, and takes a value between zero and one.

[0050] In one example, for a test model output in a key-value pair structured format, the corresponding conclusion and confidence level fields are directly read according to a preset format; for a test model output in natural language format, the values ​​of the conclusion and confidence level fields are located and mapped from the original response according to a preset pattern or keywords. Optionally, when the confidence level is not explicitly given in the original response, the value of the confidence level field is mapped from the corresponding expression in the original response.

[0051] Based on the above analysis, the diverse judgment and reasoning responses of each model under test are grouped into the same field format, and the three-level classification prediction results are uniformly extracted from the conclusion field.

[0052] In one embodiment of the present invention, before obtaining the pre-built baseline test set, the preset reference model set, and the pre-stored scores of the reference model set on historical versions of the baseline test set, the process further includes, but is not limited to: S301, performs standardized text processing on heterogeneous format files to obtain a structured text representation; Specifically, in the process of building the baseline test set, for the heterogeneous format files attached to the test cases, the corresponding parsing and conversion are performed according to their file types, the information in the files is extracted and organized into text form, and the structured text representation of the heterogeneous format file is obtained.

[0053] For example, for email files, the sender, recipient, subject, and receiving link information in the email header, as well as links in the body, are extracted, and the target address of the link is retained instead of just its displayed text when the body is in hypertext format; for traffic files, the source address, destination address, port, transport protocol, and payload summary of each data stream are extracted; for script files or executable files, the file hash, type identifier, and strings and imported function list obtained from static analysis are extracted, but their binary content is not included.

[0054] S302, determine the coverage rate of the structured text representation to the preset key field set, and the semantic retention rate of the structured text representation to the content corresponding to the key field set in the heterogeneous format file, wherein the preset key field set is a set of fields pre-defined according to the type of heterogeneous format file and used as the basis for judgment; In this embodiment, the preset key field set is set according to the type of heterogeneous format file. For example, the preset key field set corresponding to the email file includes the receiving link information in the email header and the link address in the body, and the preset key field set corresponding to the traffic file includes the destination address and the transmission protocol.

[0055] The coverage rate is the ratio of the number of fields appearing in the structured text representation within the preset key field set to the total number of fields in the preset key field set; the semantic retention rate is the ratio of the number of fields in the preset key field set whose corresponding content is fully retained within the structured text representation to the total number of fields in the preset key field set.

[0056] S303 determines the information fidelity score based on coverage and semantic retention.

[0057] Optionally, the information fidelity score is the weighted sum of coverage and semantic retention rate, or the product of coverage and semantic retention rate. The structured text representation and information fidelity score are recorded together with the corresponding test cases in the baseline test set, and are used to reduce the weight or remove test cases with information fidelity scores below a preset first threshold when determining the expected operational cost.

[0058] In one embodiment of the present invention, both the three-level classification security label and the three-level classification prediction result include a malicious category, a suspicious category, and a safe category. During security operations, alarms predicted as malicious are processed, alarms predicted as suspicious are subject to manual review, and alarms predicted as safe are disabled. The asymmetric misclassification cost matrix assigns a cost value to each combination of the true category and the predicted category; the cost is zero when the true category and the predicted category are the same, and a misclassification cost is applied when they are different.

[0059] In the asymmetric misclassification cost matrix, the cost of misclassifying a true category as malicious but predicting it as safe is greater than the cost of misclassifying a true category as malicious but predicting it as suspicious; the cost of misclassifying a true category as safe but predicting it as malicious is greater than the cost of misclassifying a true category as safe but predicting it as suspicious.

[0060] For example, the cost of misclassifying a security incident as malicious but actually malicious is the average loss value of the security incident; the cost of misclassifying a security incident as suspicious but actually malicious is the product of the average loss value and the probability that the suspicious alarm is missed during manual review; the cost of misclassifying a security incident as malicious is the sum of business interruption loss and ineffective emergency work time cost; and the cost of misclassifying a security incident as suspicious is the work time cost of manual review.

[0061] As a supplement to the expected operational costs, basic classification indicators can be determined based on the three-level classification prediction results and the three-level classification security labels. The three-level classification security labels are mapped to both malicious and suspicious categories in the three-level classification prediction results as positive, and the safe category as negative. True positives, false positives, false negatives, and true negatives are counted using a micro-average method to determine accuracy, precision, recall, and F1 score. Optionally, only the malicious category is mapped to positive to obtain the precise malicious identification rate. Optionally, the above basic classification indicators are included in the comparison criteria when determining the recommendation model.

[0062] In one embodiment of the present invention, determining the evaluation condition equivalence index based on the rate-limited retry rate, backoff waiting time, and serialization queuing time of each model under test includes, but is not limited to, the following steps: S401, determine the weighted sum based on the standard deviation of the rate-limited retry rate, the standard deviation of the backoff waiting time, and the standard deviation of the serialization queuing time; Specifically, the standard deviations of the rate-limited retry rate, the backoff waiting time, and the serialization queuing time are calculated among the various models under test. Each standard deviation reflects the degree of dispersion of the corresponding value among the various models under test. The three standard deviations are then summed with their respective weights to obtain a weighted sum.

[0063] Optionally, the weights corresponding to the three standard deviations mentioned above are equal.

[0064] S402, determine the evaluation condition equivalence index based on the normalized weighted sum, wherein the evaluation condition equivalence index decreases as the normalized weighted sum increases.

[0065] The above weighted sum is normalized to between zero and one, and the normalized weighted sum is subtracted from one to obtain the evaluation condition equivalence index between zero and one. The greater the dispersion of the above values ​​among the models under test, the larger the weighted sum, the larger the normalized weighted sum, and the smaller the evaluation condition equivalence index.

[0066] For example, normalization is achieved by weighting and dividing by a preset normalization factor.

[0067] Subsequently, the evaluation condition equivalence index is compared with a preset second threshold to determine whether the delay evaluation value should be the overall response delay or the normalized inference delay.

[0068] In one embodiment of the present invention, the test cases are further labeled with a list of key evidence; The recommendation model is determined from multiple test models based on expected operating costs, delayed evaluation values, and equivalent alignment scores, including but not limited to the following steps: S501, Extract the chain of reasoning evidence containing evidence type, evidence value and relevance from the judgment and reasoning response; In this embodiment, the evidence given by the model under test during the judgment is extracted from the judgment and reasoning response of the model under test to the alarm corresponding to the test case, and a reasoning evidence chain consisting of multiple evidence items is obtained. Each evidence item includes evidence type, evidence value and correlation degree. The correlation degree indicates the degree of relevance of the evidence item to the judgment conclusion.

[0069] For example, types of evidence include URL reputation, IP threat intelligence, file hashes, domain information, email header anomalies, and traffic patterns.

[0070] S502, the evidence items in the inference evidence chain that match the evidence type and evidence value with the key evidence list are weighted and summed according to their relevance to obtain the evidence chain matching degree; In this embodiment, the key evidence list of the test case is a pre-annotated set of key evidence items on which the judgment conclusion of the test case is based. The evaluation device compares each evidence item in the reasoning evidence chain with the key evidence list one by one, and determines the evidence items whose evidence type and evidence value both correspond to a certain key evidence item in the key evidence list as matching evidence items. The device then weights and sums the correlation of each matching evidence item to obtain the evidence chain matching degree of the model under test for the test case.

[0071] S503 determines the recommendation model from multiple test models based on expected operating costs, delayed evaluation values, equivalent alignment scores, and evidence chain matching degrees.

[0072] When determining the recommendation model, in addition to expected operating costs, delayed evaluation values, and equivalent alignment scores, the degree of evidence chain matching is further incorporated into the comparison criteria.

[0073] For example, the expected operating cost, delayed evaluation value, equivalent alignment score and evidence chain matching degree of each model under test are normalized and then weighted and summed to obtain the comprehensive evaluation of each model under test, and the recommended model is determined accordingly.

[0074] As a supplement to the evidence chain matching degree, the key evidence coverage can be further determined based on the determination of the evidence chain matching degree. Each key evidence item in the key evidence list is marked with a degree of importance; key evidence items with an importance degree higher than a set value are identified as core evidence items. The key evidence coverage is calculated as the ratio of the number of core evidence items in the reasoning evidence chain that match the core evidence items (i.e., the evidence type and evidence value are identical) to the total number of core evidence items in the key evidence list. Optionally, when determining the recommendation model, key evidence coverage can be further incorporated into the comparison criteria.

[0075] As another possible implementation method for determining the recommendation model in step S107 above, if the test cases are also labeled with threat severity levels, the severity-weighted recall rate can also be included in the recommendation considerations.

[0076] S601. Determine the recall rate of each model under test according to the severity level, and calculate the severity-weighted recall rate by weighting the recall rates.

[0077] The threat severity level of a test case is used to represent the threat level of the alarm corresponding to that test case, and for example, it includes four levels: severe, high, medium, and low. The evaluation device uses test cases with the three-level classification security label as malicious or suspicious as positive samples, groups the test cases according to the threat severity level, determines the recall rate of each model under test at each threat severity level, and then assigns weights to each threat severity level. The severity-weighted recall rate of the model under test is obtained by weighting the recall rate of each threat severity level with its weight.

[0078] The higher the threat severity level, the greater the corresponding weight. For example, the weights for the four levels of severe, high, medium, and low are 10, 5, 2, and 1, respectively; optionally, the above weights can be configured according to actual needs.

[0079] S602 determines the recommendation model from multiple test models based on expected operating costs, delay evaluation values, equivalent alignment scores, and severity-weighted recall.

[0080] When determining the recommendation model, in addition to expected operating costs, delayed evaluation values, and equivalent alignment scores, severity-weighted recall rate is further incorporated into the comparison criteria.

[0081] For example, the expected operating cost, delay evaluation value, equivalent alignment score and severity-weighted recall of each model to be tested are normalized and then weighted and summed to obtain a comprehensive evaluation, and the recommendation model is determined accordingly.

[0082] As another possible implementation method for determining the recommendation model in step S107 above, the computing cost dimension of each model to be tested can be introduced, and the optimal choice can be made between computing cost and evaluation quality. Specifically, the following processing is included.

[0083] S701 determines the computing power cost of a single alarm for each model under test based on the token consumption of each model under test.

[0084] Based on the token consumption of each model under test, the input token and output token are distinguished, and then multiplied by the corresponding input unit price and output unit price respectively, and summed to obtain the computing power cost of the model under test for processing a single alarm.

[0085] Since the billing methods for each model under test differ, the unit price is normalized. Optionally, for models billed by token, the token usage returned by their interface and the published unit price are directly used; for models billed by character, the mapping relationship between the number of characters and the number of tokens is determined using a calibrated corpus and the model's word segmenter, and a piecewise linear mapping function is obtained by fitting the text length segments, thereby converting the number of characters into token consumption; for models billed by the number of requests, the equivalent unit price is obtained by dividing the price per request by the average token consumption of the model on the baseline test set; for models billed by GPU time, the inference time is converted to the equivalent inference time of the reference GPU, and then combined with the time price to obtain the computing power cost.

[0086] For example, the calibration corpus contains 500 to 1,000 security assessment texts, categorized into short texts, Chinese texts, and long texts, with Chinese texts and English texts each accounting for half. Optionally, the conversion method used for the above normalization, the version of the calibration corpus, the conversion coefficients, and the error range are also included in the evaluation report.

[0087] S702 uses the cost of computing power per alarm as the cost dimension and the quality of assessment as the effect dimension to construct cost-benefit scatter points for each model under test, and then obtains the Pareto front of these cost-benefit scatter points.

[0088] Among them, the quality assessment is a score obtained by combining expected operating cost, delay evaluation value and equivalent alignment score; the Pareto front is the set of scattered points corresponding to each model under test, where there is no other scattered point with lower computing cost and higher assessment quality.

[0089] For example, the Pareto front is obtained as follows: each model under test is arranged in ascending order of the computing power cost of a single alarm and scanned sequentially. The maximum judgment quality value among the scanned scatter points is maintained. When the judgment quality value of a scatter point is greater than the maximum value, it is added to the Pareto front and the maximum value is updated; otherwise, it is skipped.

[0090] Optionally, a model combination consisting of at least two test models is included as a virtual scatter point in the above Pareto frontier calculation. In this model combination, the first test model is used for initial screening and the second test model is used for verification. Its computing cost is the computing cost of the first test model plus the product of the computing cost of the second test model and the probability that the initial screening result needs to be verified. Its evaluation quality is determined by the end-to-end evaluation results of the model combination on the baseline test set.

[0091] S703 determines the recommended model based on constraints at the Pareto front.

[0092] Given a computing cost constraint, the model with the highest evaluation quality among the scattered points on the Pareto front where the computing cost does not exceed the constraint is selected as the recommended model; given an evaluation quality constraint, the model with the lowest computing cost among the scattered points on the Pareto front where the evaluation quality is not lower than the constraint is selected as the recommended model.

[0093] In one embodiment of the present invention, the anchored test case identifier is set according to a difficulty stability index, which is determined based on the ratio of the standard deviation to the mean of the reasoning scores of the reference model set on the corresponding test case; wherein, the anchored test case identifier of the test case whose difficulty stability index is higher than a set threshold is set to true.

[0094] During the construction of the baseline test set, for each test case, the reference model set performs judgment and reasoning multiple times on the test case and records the score for each time; calculate the standard deviation and mean of these scores, and subtract the ratio of the standard deviation and the mean from one to obtain the difficulty stability index of the test case. This index reflects the stability of the reference model set's score on the test case, and the smaller the score fluctuation, the larger the index.

[0095] The anchor test case flags of test cases whose difficulty stability index is higher than the set threshold are set to true, and the anchor test case flags of the remaining test cases are set to false; the test cases with the anchor test case flags set to true constitute the anchor test case set used to determine the equivalent alignment score in S105.

[0096] Optionally, the above multiple judgments and inferences refer to at least three judgments and inferences performed on the reference model set; for example, the threshold is set to 0.85.

[0097] Another embodiment of the present invention relates to a multi-model recommendation device for security alarm assessment, such as... Figure 2 As shown, the device includes: The acquisition module 20 is used to acquire a pre-built baseline test set, a preset reference model set, and the pre-stored scores of the reference model set on the historical version test set of the baseline test set; wherein, the baseline test set includes multiple test cases, each test case has a three-level classification security label and an anchor test case identifier, the heterogeneous format files in the test cases are structured text representations, and each test case carries a pre-calculated information fidelity score; The scheduling module 21 is used to schedule multiple test models with different architectures to perform judgment and reasoning on the baseline test set; The first determining module 22 is used to determine the evaluation condition equivalence index based on the rate limit retry rate, backoff waiting time and serialized queuing time of each model under test during the scheduling execution process, and to subtract the backoff waiting time and the serialized queuing time from the overall response delay of each model under test to obtain the normalized inference delay. The second determining module 23 is used to extract the three-level classification prediction results from the judgment and reasoning response of each model to be tested, and to determine the expected operating cost based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security label and the preset asymmetric misclassification cost matrix, wherein the costs corresponding to at least two types of misclassification in the asymmetric misclassification cost matrix are not equal, and when determining the expected operating cost, test cases with information fidelity scores lower than the preset first threshold are downweighted or eliminated. Alignment module 24 is used to determine the equivalent alignment score of the model under test on the historical version test set based on the average score of the reference model set on the anchoring example set of the current version of the baseline test set, the pre-stored score, and the score of the model under test on the anchoring example set of the current version. The anchoring example set is the set of test cases in the baseline test set whose anchoring test cases are identified as true. Evaluation module 25 is used to determine the delay evaluation value of each model under test, wherein when the evaluation condition equivalence index is not lower than a preset second threshold, the overall response delay is used as the delay evaluation value, and when the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference delay is used as the delay evaluation value. Recommendation module 26 is used to determine a recommendation model from the plurality of models to be tested based on the expected operating cost, the delay evaluation value and the equivalent alignment score.

[0098] Optionally, in this embodiment, the scheduling module 21 includes: The first processing submodule is configured to, for any one of the plurality of test models, convert preset evaluation prompts into input prompts adapted to the interface format of the test model according to the input template corresponding to that test model, and input them into the test model; and, The second processing submodule is used to parse the judgment and reasoning response output by the model under test into a conclusion field and a confidence field in a preset format, wherein the conclusion field includes the three-level classification prediction result.

[0099] Optionally, in this embodiment, it further includes: The text processing module is used to perform normalized text processing on the heterogeneous format file to obtain the structured text representation before obtaining the pre-built baseline test set, the preset reference model set, and the pre-stored scores of the reference model set on the historical version test set of the baseline test set. The third determining module is used to determine the coverage rate of the structured text representation to the preset key field set, and the semantic retention rate of the structured text representation to the content corresponding to the key field set in the heterogeneous format file, wherein the preset key field set is a set of fields pre-defined according to the type of the heterogeneous format file and used as the basis for judgment. The fourth determining module is used to determine the information fidelity score based on the coverage rate and the semantic retention rate.

[0100] Optionally, in this embodiment, both the three-level classification security label and the three-level classification prediction result include a malicious category, a suspicious category, and a safe category; wherein, in the asymmetric misclassification cost matrix, the cost corresponding to the misclassification that the true category is malicious but is predicted as safe is greater than the cost corresponding to the misclassification that the true category is malicious but is predicted as suspicious; the cost corresponding to the misclassification that the true category is safe but is predicted as malicious is greater than the cost corresponding to the misclassification that the true category is safe but is predicted as suspicious.

[0101] Optionally, in this embodiment, the first determining module 22 includes: The first determining submodule is used to determine a weighted sum based on the standard deviation of the rate-limited retry rate, the standard deviation of the backoff waiting time, and the standard deviation of the serialization queuing time; The second determining submodule is used to determine the evaluation condition equivalence index based on the normalized weighted sum, wherein the evaluation condition equivalence index decreases as the normalized weighted sum increases.

[0102] Optionally, in this embodiment, the test cases are further labeled with a list of key evidence; wherein, the recommendation module 26 includes: The acquisition submodule is used to extract a chain of reasoning evidence containing evidence type, evidence value and relevance from the judgment and reasoning response; The third processing submodule is used to sum the evidence items in the inference evidence chain that match the evidence type and evidence value with the key evidence list according to the correlation degree to obtain the evidence chain matching degree. The third determination submodule is used to determine the recommendation model from the multiple test models based on the expected operating cost, the delay evaluation value, the equivalent alignment score, and the evidence chain matching degree.

[0103] Optionally, in this embodiment, the anchored test case identifier is set according to a difficulty stability index, which is determined based on the ratio of the standard deviation to the mean of the inference scores of the reference model set on the corresponding test case; wherein, the anchored test case identifier of the test case whose difficulty stability index is higher than a set threshold is set to true.

[0104] Another embodiment of the present invention relates to an electronic device, such as Figure 3 As shown, it includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, the instructions being executed by the at least one processor 301 to enable the at least one processor 301 to execute the multi-model recommendation method for security alarm assessment in the above embodiments.

[0105] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein.

[0106] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0107] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0108] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-model recommendation method for security alarm analysis, characterized in that, include: Obtain a pre-built baseline test set, a preset reference model set, and the pre-stored scores of the reference model set on historical versions of the baseline test set; wherein, the baseline test set includes multiple test cases, each test case has a three-level classification security label and an anchor test case identifier, the heterogeneous format files in the test cases are structured text representations, and each test case carries a pre-calculated information fidelity score; Multiple test models with different architectures are scheduled to perform analytical reasoning on the baseline test set; Based on the rate-limited retry rate, backoff waiting time, and serialized queuing time of each model under test, an evaluation condition equivalence index is determined; and the backoff waiting time and serialized queuing time are subtracted from the overall response delay of each model under test to obtain the normalized inference delay. The three-level classification prediction results are extracted from the judgment and reasoning responses of each model under test. Based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security labels, and the preset asymmetric misclassification cost matrix, the expected operational cost is determined. In the asymmetric misclassification cost matrix, the costs corresponding to at least two types of misclassification are not equal. When determining the expected operational cost, test cases with information fidelity scores lower than a preset first threshold are downweighted or eliminated. Based on the average score of the reference model set on the anchoring example set of the current version of the baseline test set, the pre-stored score, and the score of the model under test on the anchoring example set of the current version, the equivalent alignment score of the model under test on the historical version test set is determined; wherein, the anchoring example set is the set of test cases in the baseline test set whose anchoring test cases are identified as true. Determine the delay evaluation value for each model under test, wherein when the evaluation condition equivalence index is not lower than a preset second threshold, the overall response delay is used as the delay evaluation value, and when the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference delay is used as the delay evaluation value. Based on the expected operating cost, the delay evaluation value, and the equivalent alignment score, a recommendation model is determined from the plurality of models to be tested.

2. The method according to claim 1, characterized in that, The scheduling of multiple test models with different architectures to perform analytical reasoning on the baseline test set includes: For any of the multiple test models, according to the input template corresponding to that test model, the preset evaluation prompts are converted into input prompts adapted to the interface format of that test model, and then input into that test model; and, The judgment and reasoning response output by the model under test is parsed into a conclusion field and a confidence field in a preset format, wherein the conclusion field includes the three-level classification prediction result.

3. The method according to claim 1, characterized in that, Before obtaining the pre-built baseline test set, the preset reference model set, and the pre-stored scores of the reference model set on historical versions of the baseline test set, the method further includes: The heterogeneous format file is subjected to normalized text processing to obtain the structured text representation; The coverage rate of the structured text representation to a preset set of key fields and the semantic retention rate of the structured text representation to the content corresponding to the set of key fields in the heterogeneous format file are determined, wherein the preset set of key fields is a set of fields pre-defined according to the type of the heterogeneous format file and used as the basis for the judgment. The information fidelity score is determined based on the coverage rate and the semantic retention rate.

4. The method according to claim 1, characterized in that, Both the three-level classification security label and the three-level classification prediction results include malicious category, suspicious category, and safe category; among which... In the asymmetric misclassification cost matrix, the cost of misclassifying a true category as malicious but predicting it as safe is greater than the cost of misclassifying a true category as malicious but predicting it as suspicious; the cost of misclassifying a true category as safe but predicting it as malicious is greater than the cost of misclassifying a true category as safe but predicting it as suspicious.

5. The method according to claim 1, characterized in that, The evaluation condition equivalence index is determined based on the rate-limited retry rate, backoff waiting time, and serialized queuing time of each model under test, including: The weighted sum is determined based on the standard deviation of the rate-limited retry rate, the standard deviation of the backoff waiting time, and the standard deviation of the serialization queuing time; The evaluation condition equivalence index is determined based on the normalized weighted sum, wherein the evaluation condition equivalence index decreases as the normalized weighted sum increases.

6. The method according to claim 1, characterized in that, The test cases are also labeled with a list of key evidence; wherein, determining the recommendation model from the multiple models to be tested based on the expected operating cost, the delay evaluation value, and the equivalent alignment score includes: Extract the chain of reasoning evidence, which includes evidence type, evidence value, and relevance, from the analytical reasoning response; The evidence items in the inference evidence chain that match the evidence type and evidence value with the key evidence list are weighted and summed according to the relevance to obtain the evidence chain matching degree. Based on the expected operating cost, the delay evaluation value, the equivalent alignment score, and the evidence chain matching degree, a recommendation model is determined from the plurality of models to be tested.

7. The method according to claim 1, characterized in that, The anchored test case identifier is set according to the difficulty stability index, which is determined by the ratio of the standard deviation to the mean of the reasoning scores of the reference model set on the corresponding test cases. In this context, the anchor test case identifier for test cases whose difficulty stability index exceeds a set threshold is set to true.

8. A multi-model recommendation device for security alarm analysis, characterized in that, include: The acquisition module is used to acquire a pre-built baseline test set, a preset reference model set, and the pre-stored scores of the reference model set on historical versions of the baseline test set; wherein, the baseline test set includes multiple test cases, each test case has a three-level classification security label and an anchor test case identifier, the heterogeneous format files in the test cases are structured text representations, and each test case carries a pre-calculated information fidelity score; The scheduling module is used to schedule multiple test models with different architectures to perform judgment and reasoning on the baseline test set; The first determining module is used to determine the evaluation condition equivalence index based on the rate limit retry rate, backoff waiting time and serialized queuing time of each model under test during the scheduling execution process, and to subtract the backoff waiting time and the serialized queuing time from the overall response delay of each model under test to obtain the normalized inference delay. The second determining module is used to extract the three-level classification prediction results from the judgment and reasoning response of each model to be tested, and to determine the expected operating cost based on the three-level classification prediction results, the confusion probability distribution of the three-level classification security label and the preset asymmetric misclassification cost matrix. In the asymmetric misclassification cost matrix, the costs corresponding to at least two types of misclassification are not equal, and when determining the expected operating cost, test cases with information fidelity scores lower than the preset first threshold are downweighted or eliminated. The alignment module is used to determine the equivalent alignment score of the model under test on the historical version test set based on the average score of the reference model set on the anchoring example set of the current version of the baseline test set, the pre-stored score, and the score of the model under test on the anchoring example set of the current version. The anchoring example set is the set of test cases in the baseline test set whose anchoring test cases are identified as true. An evaluation module is used to determine the delay evaluation value of each model under test. When the evaluation condition equivalence index is not lower than a preset second threshold, the overall response delay is used as the delay evaluation value. When the evaluation condition equivalence index is lower than the preset second threshold, the normalized inference delay is used as the delay evaluation value. The recommendation module is used to determine a recommended model from the plurality of models to be tested based on the expected operating cost, the delay evaluation value, and the equivalent alignment score.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multi-model recommendation method for security alarm assessment as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-model recommendation method for security alarm assessment as described in any one of claims 1 to 7.

Images