Automated software vulnerability assessment based on generative artificial intelligence

Abstract

A computer-implemented method is provided for receiving vulnerability reports of software. The vulnerability report specifies vulnerable libraries used by the software and the paths of these vulnerable libraries within the software. The method can generate a summary of the vulnerability report using a generative artificial intelligence (AI) model, retrieve a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database, generate a summary of the relevant vulnerability report sets using the generative AI model, generate multiple preliminary decisions regarding the validity of the vulnerability report based on the vulnerability report summary and the summaries of the relevant vulnerability report sets, and generate a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions. Related systems and software for implementing this method are also disclosed.

Description

Technical Field

[0001] This disclosure relates to automated software vulnerability assessment based on generative artificial intelligence. Background Technology

[0002] Software development, particularly open-source software (OSS), has driven progress in transparency, innovation, and community collaboration, providing substantial benefits to the software industry. However, it has also brought challenges, especially the increasing discovery and disclosure of vulnerabilities within software. Software vulnerabilities are security weaknesses that, if exploited, can compromise systems and data. Existing technologies aim to automatically detect these vulnerabilities (e.g., generate vulnerability reports), but often struggle to accurately assess their effectiveness, frequently resulting in false positives. This leads to significant resource waste, as developers must manually evaluate a large number of invalid vulnerability reports. Therefore, there is room for improvement in methods to enhance the accuracy and efficiency of vulnerability assessment. Summary of the Invention

[0003] In one embodiment, a computing system is provided, including: a memory; one or more hardware processors coupled to the memory; and one or more computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations including: receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report sets using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report based on the summary of the vulnerability report and the summaries of the relevant vulnerability report sets using the generative AI model; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions.

[0004] In another embodiment, a computer-implemented method is provided, comprising receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report sets using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report based on the summary of the vulnerability report and the summaries of the relevant vulnerability report sets using the generative AI model; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions.

[0005] In another embodiment, one or more non-transitory computer-readable media are provided, having computer-executable instructions encoded thereon, which, when executed by a computing system, cause the computing system to perform a method, the method comprising: receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report sets using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report using the generative AI model based on the summary of the vulnerability report and the summaries of the relevant vulnerability report sets; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions. Attached Figure Description

[0006] Figure 1 This is a block diagram depicting the overall framework of the automated software vulnerability assessment techniques disclosed in this paper.

[0007] Figure 2 This is a block diagram depicting an example automated software vulnerability assessment system.

[0008] Figure 3 This is a flowchart illustrating an example overall method for automating software vulnerability assessment.

[0009] Figure 4 This is an architecture diagram of an example large language model.

[0010] Figure 5 It is a diagram illustrating the entire process of extracting expert knowledge related to the vulnerability database.

[0011] Figure 6 This is a flowchart depicting an example process of retrieving a set of relevant defect reports from a defect report database.

[0012] Figure 7 Example experimental results are shown.

[0013] Figure 8 This is a block diagram of an example computing system in which the described embodiments can be implemented.

[0014] Figure 9 This is a block diagram of an example cloud computing environment that can be used in conjunction with the technologies described in this article. Detailed Implementation

[0015] Software Vulnerability Assessment Overview

[0016] In today's interconnected digital environment, detecting software vulnerabilities is a critical component of maintaining secure, stable, and reliable systems. Vulnerabilities include not only security weaknesses or flaws that could expose a system to unauthorized access or malicious exploitation, but also other defects, such as software bugs, that can lead to crashes, performance degradation, or other undesirable behaviors. These issues can disrupt business operations, harm user experience, and undermine trust in software systems. As the software ecosystem becomes increasingly complex, particularly with the growing reliance on open-source software (OSS), timely identification and mitigation of such vulnerabilities are crucial for ensuring operational continuity and data integrity. The widespread adoption of OSS libraries further amplifies this need, as their public availability increases the likelihood that malicious and unintentional defects will be discovered and exploited, or cause unintended disruptions.

[0017] Existing technologies for automated detection of software vulnerabilities rely heavily on databases that serve as repositories for discovered vulnerabilities. These databases, such as the National Vulnerability Database (NVD) maintained by the National Institute of Standards and Technology (NIST), provide a centralized source of information on identified vulnerabilities, including unique identifiers, severity scores, affected software components and versions, and detailed vulnerability descriptions. For example, the NVD uses a Common Vulnerability and Exposure (CVE) system to catalog vulnerabilities, thereby facilitating vulnerability identification and remediation across the software ecosystem.

[0018] Building upon this foundation, current vulnerability detection technologies utilize various methods to scan and analyze vulnerabilities in codebases. For example, Named Entity Recognition (NER)-based methods extract entities from vulnerability descriptions in these repositories and match them with software library names, thereby establishing a direct association between vulnerabilities and specific components. As another example, Extreme Multi-Label Learning (XML)-based methods apply artificial intelligence (AI) or machine learning classifiers trained on labeled datasets to map vulnerability descriptions to their corresponding libraries. These technologies form the backbone of existing open-source tools such as OWASP DependencyCheck and Retire.js, as well as commercial tools such as Mend (formerly known as WhiteSource) and Black Duck Binary Analysis (BDBA), which aim to identify known vulnerabilities in software systems and generate vulnerability reports.

[0019] Despite these advances, significant drawbacks remain. Current vulnerability detection tools typically rely on metadata associated with OSS repositories and vulnerability descriptions, which is often incomplete, inconsistent, or outdated. This leads to a high false-positive rate, where vulnerabilities are reported inaccurately, requiring substantial manual work for verification and remediation. Furthermore, the inherent complexity and dependencies of modern software systems exacerbate these problems, making it challenging to distinguish between valid and invalid software vulnerability reports. For example, historical analysis of vulnerability reports in large-scale software development environments shows that over 96% of reported software vulnerabilities may be invalid, resulting in wasted resources and delays in remediation efforts.

[0020] The techniques described in this paper address many of the aforementioned challenges by implementing an automated software vulnerability assessment system that leverages generative artificial intelligence (GenAI) to validate vulnerability reports generated by existing software vulnerability detection tools. By distinguishing between valid (real) and invalid (false) vulnerabilities, the disclosed system reduces wasted resources on analyzing defect reports, enabling teams to focus on genuine vulnerabilities and enhance security and efficiency. While OSS is used as an illustrative example, the disclosed techniques are equally applicable to non-open-source software.

[0021] Example framework for automated software vulnerability assessment

[0022] Figure 1 An overall framework 100 for automated software vulnerability assessment based on the techniques disclosed herein is described.

[0023] Within framework 100, vulnerability detection 110 involves using existing tools 116 (e.g., BDBA, etc.) to detect vulnerabilities in software 112. Software 112 may include methods or functions 114 that are part of a vulnerable library and have been identified as vulnerable according to an existing vulnerability database (e.g., NVD, etc.). Tool 116 may scan software 112 and generate one or more vulnerability reports 102 in a specific format. Each vulnerability report 102 may include details of at least one detected software vulnerability, such as the identified vulnerable library and affected version, the path of the vulnerable library in the software codebase, related symptoms, and optional other relevant information (e.g., suggested solutions or mitigations, examples, etc.).

[0024] Typically, vulnerability reports 102 generated by tool 116 are sent directly to ticket processing program 104, generating a "ticket" for troubleshooting. Ticket processing program 104 is a centralized bug tracking or ticketing system (e.g., Bugzilla), where technical support team 106 can evaluate reports 102 and assess vulnerabilities in software 112. However, this process is inefficient and wasteful of resources due to the large number of invalid (erroneous) reports 102 (e.g., for complex software development, tool 116 may generate hundreds or thousands of vulnerability reports per day), as support team 106 must manually review and eliminate non-existent vulnerabilities, delaying the resolution of real vulnerabilities.

[0025] According to the disclosed technology, instead of being sent to the ticket processing program 104, vulnerability report 102 is directed to an automated software vulnerability assessment system 120, which includes four components: a vulnerability knowledge distiller 130, an expert knowledge distiller 140, an auto-suggestion generator 150, and an adjudicator 160. The automated software vulnerability assessment system 120 can communicate with a GenAI model 180, which can be externally hosted, such as on a third-party platform. Alternatively, the GenAI model 180 can be deployed locally, for example, on the automated software vulnerability assessment system 120.

[0026] Each component (130, 140, 150, and 160) can be powered by the GenAI model 180. Specifically, the vulnerability knowledge extractor 130 is configured to use the GenAI model 180 to automatically extract information from the vulnerability report 102 in real time and generate a vulnerability summary 132.

[0027] Expert knowledge extractor 140 is configured to automatically retrieve relevant defect report sets in real time from defect report database 170 (also known as ticket database), which specifies vulnerable libraries identified by vulnerability report 102. Defect report database 170 may store a collection of defect reports (or tickets) previously generated by support team 106. Each defect report may include a description, diagnosis, solution, and other analysis of a vulnerability report or ticket generated by tool 116 when scanning historical software (which may be the same software 112 or different software). The retrieved relevant defect reports provide domain-specific insights from relevant historical data (e.g., discussions, analyses, and assessments of similar past vulnerability reports by support team 106). Expert knowledge extractor 140 can then use GenAI model 180 to automatically generate a summary 142 (also known as a "ticket summary") of the relevant defect report set in real time. Figure 1The defect report database 170 inside the expert knowledge extractor 140 is shown, but in other examples, the defect report database 170 may be located outside the expert knowledge extractor 140 (e.g., outside the automated software vulnerability assessment system 120).

[0028] Based on a predefined thinking paradigm 154 (also known as a “mindset”), the auto-prompt generator 150 is configured to use the GenAI model 180 to create multiple muted prompts 156 (also known as derived prompts or refinements) from a base prompt (also known as a meta-prompt 152). As described more fully below, each muted prompt 156 provides a different perspective from the meta-prompt 152 in guiding the generative behavior of the GenAI model 180.

[0029] Arbitrator 160 can utilize the results of the other three components to determine the validity of vulnerability report 102. Specifically, arbitrator 160 is configured to automatically generate multiple prompts 162 based on multiple derived prompts 156 and optional meta-prompts 152. Each prompt 162 may contain a vulnerability summary 132 (generated by vulnerability knowledge extractor 130) and a summary 142 of the relevant vulnerability report set (generated by expert knowledge extractor 140). Using multiple prompts 162, arbitrator 160 can individually prompt GenAI model 180 to generate multiple preliminary decisions 164 regarding the validity of vulnerability report 102 in real time. Arbitrator 160 can then use a majority voting mechanism (e.g., calculating a weighted sum of the multiple preliminary decisions 164) to obtain a final decision 166 regarding the validity of vulnerability report 102.

[0030] The final decision 166, which can indicate whether vulnerability report 102 is valid or invalid (and in some cases is uncertain), can then be forwarded to ticket processing procedure 104 for support team 106 to perform further analysis.

[0031] Therefore, framework 100 can significantly reduce the burden on support team 106 by filtering out fake vulnerabilities early in the process, for example, by not generating work orders for vulnerability reports that are determined to be invalid.

[0032] Example of an overall automated software vulnerability assessment system

[0033] Figure 2 The diagram shows the overall block diagram of an automated software vulnerability assessment system 220, which can be... Figure 1 Example embodiment of the automated software vulnerability assessment system 120.

[0034] System 220 is configured to (at runtime) automatically determine the validity of software vulnerability reports 202. As described above, vulnerability reports 202 can be generated by another tool designed to detect software vulnerabilities based on existing vulnerability databases (e.g., NVD, etc.), and vulnerability reports 202 can identify vulnerable libraries used by the software and the paths to vulnerable libraries within the software. Figure 2 As shown, the automated software vulnerability assessment system 220 includes a parser 255, an AI agent 250, a hint generator 240, a search engine 275, and an adjudicator 260, each of which can be implemented as a software module, a hardware module, a firmware module, or a combination thereof.

[0035] AI Agent 250 and GenAI Model 280 (similar to) Figure 1 Communication with the GenAI model 280. For example, the AI ​​agent 250 can send prompts to the GenAI model 280 and receive responses from the GenAI model 280. The prompt generator 240 can be configured to generate various prompts 245 at runtime based on different prompt templates 235 maintained by the system 220.

[0036] AI agent 250 can be configured to coordinate various interactions between system 220 and GenAI model 280, including, for example, extracting vulnerability knowledge from vulnerability report 202 and from vulnerability report database 270 (similar to...). Figure 1 The database of defect reports (170) extracts expert knowledge from relevant defect reports or work orders, generates multiple derivative prompts based on meta prompts, and generates multiple preliminary decisions regarding the validity of vulnerability report 202, as described above.

[0037] For example, upon receiving vulnerability report 202, parser 255 can extract various fields from vulnerability report 202 and provide the extracted field information to AI agent 250. AI agent 250 can instruct prompt generator 240 to generate a vulnerability knowledge extraction prompt based on a prompt template 235 at runtime, which contains the field information extracted by parser 255. The generated prompt may include instructions for GenAI model 280 to extract vulnerability information from vulnerability reports 202 and the vulnerability libraries used by the software. AI agent 250 can then prompt GenAI model 280 with the vulnerability knowledge extraction prompt and receive a response from GenAI model 280 providing a vulnerability summary 232 of vulnerability report 202.

[0038] AI agent 250 can also instruct search engine 275 to retrieve a set of relevant vulnerability reports from vulnerability report database 270 specifying vulnerable libraries identified by vulnerability report 202. Additional details for retrieving relevant vulnerability reports are described more fully below. AI agent 250 can then instruct prompt generator 240 to generate an expert knowledge refinement prompt based on another prompt template 235 at runtime. The generated expert knowledge refinement prompt includes instructions for GenAI model 280 to extract a work order summary 242 or summary of the relevant vulnerability report set from the relevant vulnerability report set. AI agent 250 can then prompt GenAI model 280 with the expert knowledge refinement prompt and receive a response from GenAI model 280 providing a work order summary 242 or summary of the relevant vulnerability report set.

[0039] In some examples, at least one of the prompt templates 235 can be a meta-prompt. AI agent 250 can use auto-prompt engineering techniques to derive multiple derived prompts from the meta-prompt. Specifically, AI agent 250 can instruct prompt generator 240 to generate multiple derived prompts based on another of the prompt templates 235. Each derived prompt includes instructions for GenAI model 280 to generate the derived prompt based on the meta-prompt using a specific thought paradigm or mode of thinking. AI agent 250 can then prompt GenAI model 280 with each derived prompt and receive, in response, a corresponding derived prompt generated by GenAI model 280. The multiple derived prompts generated by GenAI model 280 can also be stored as prompt templates 235 (such as meta-prompts).

[0040] Furthermore, the AI ​​agent 250 can instruct the prompt generator 240 to generate multiple decision prompts using multiple derived prompts and meta-prompts as prompt templates. In other words, decision prompts can be generated from meta-prompts or from a derived prompt representing a specific way of thinking. Each decision prompt includes instructions for the GenAI model 280 to generate a preliminary decision 264 regarding the validity of the vulnerability report 202 based on the aforementioned vulnerability summary 232 and ticket summary 242. Each preliminary decision 264 can produce a binary classification of the vulnerability report 202: valid or invalid. Additionally, each preliminary decision 264 can include supporting evidence or reasoning information for the classification, for example, instructing the GenAI model 280 to rely on which and how much relevant vulnerability reports to generate a valid or invalid classification result.

[0041] Then, the adjudicator 260 can generate a final decision 266 based on multiple preliminary decisions 264. In some examples, the adjudicator 260 can employ a majority voting mechanism, for example, by calculating a weighted sum of multiple preliminary decisions. In some examples, the weight of each preliminary decision can be determined based on how many relevant defect reports are found in the relevant defect report set by the GenAI model 280 to support the preliminary decision. In some examples, the final decision 266 can indicate whether the vulnerability report 202 is valid, invalid, or uncertain. In some examples, vulnerability reports deemed invalid will not generate a support ticket, while vulnerability reports deemed valid or uncertain can be sent to the support ticket processing procedure for further investigation.

[0042] In practice, the systems illustrated in this paper (such as automated software vulnerability assessment systems 120 and 220) can vary in complexity, incorporating additional functionalities, more complex components, etc. For example, additional functionalities may exist within system 120 or 220. Additional components may be included to achieve security, redundancy, load balancing, report design, data logging, etc.

[0043] The described computing system can be networked via wired or wireless network connections, including the Internet. Alternatively, the system can be connected via an intranet connection (e.g., in a corporate environment, government environment, etc.).

[0044] System 120 or 220, and any other system described herein, can be implemented in conjunction with any of the hardware components described herein, such as the computing systems described below (e.g., processing units, memory, etc.). In any example herein, vulnerability reports, defect reports, hint templates, hints, vulnerability summaries, ticket summaries, etc., can be stored in one or more computer-readable storage media or computer-readable storage devices. The techniques described herein are applicable to the details of operating systems or hardware and can be applied in a variety of environments to leverage the described features.

[0045] Example Overall Methodology for Automated Software Vulnerability Assessment

[0046] Figure 3 This is a flowchart illustrating an example overall method 300 for performing automated software vulnerability assessment. Method 300 can be performed, for example, by an automated software vulnerability assessment system 120 or 220.

[0047] At step 310, the method receives a vulnerability report for the selected software. The vulnerability report specifies the vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software.

[0048] At step 320, the method can generate a summary of the vulnerability report at runtime using a GenAI model (e.g., 180 or 280).

[0049] In some examples, the generated vulnerability report summary includes generating vulnerability knowledge extraction hints based on a first hint template, and using the vulnerability knowledge extraction hints to prompt a generative AI model. The first hint template may include instructions for the GenAI model to extract vulnerability information from the vulnerability report using a vulnerability library used by the selected software.

[0050] At step 330, the method can automatically retrieve a set of relevant defect reports for a specified vulnerable library from a defect report database (e.g., 170 or 270).

[0051] In some examples, the retrieval action includes searching a defect report database to identify a collection of defect reports for one or more historical software programs, and identifying relevant defect report sets from the collection of defect reports. In various examples, the defect reports of historical software specify the vulnerability libraries used by the historical software and the paths to those libraries within the historical software.

[0052] In some examples, the identification action includes comparing the paths of vulnerable libraries within the selected software with the paths of vulnerability libraries within historical software.

[0053] In some examples, the identification action also includes adding the vulnerability report of the historical software to the relevant vulnerability report set if (i) the path of the vulnerability library in the historical software contains the path of the vulnerable library in the selected software, and (ii) the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number.

[0054] In response to determining, after evaluating a set of vulnerability reports, that the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number, the method can iteratively compare the subpaths of the selected vulnerable libraries within the software with the paths of the vulnerability libraries within the historical software. In various examples, the subpaths can be derived by removing prefixes from the paths of the selected vulnerable libraries within the software.

[0055] Additional details on example methods for retrieving relevant defect report sets are provided in Figure 6 It is shown in the figure and described further below.

[0056] At step 340, the method can use the GenAI model at runtime to generate a summary of the relevant defect report set.

[0057] In some examples, generating a summary may include generating expert knowledge extraction prompts based on a second prompt template, and using these expert knowledge extraction prompts to prompt the GenAI model. The second prompt template may include instructions for the GenAI model to extract summaries and validity classifications of relevant defect report sets.

[0058] At step 350, the method may use a GenAI model to generate several preliminary decisions regarding the validity of the vulnerability report based on a summary of the vulnerability report and a summary of the relevant vulnerability report set.

[0059] In some examples, to generate multiple initial decisions, the method can use the GenAI model to obtain multiple derived cues previously derived from the meta-cues based on multiple predefined thinking paradigms. The method can then individually prompt the GenAI model using these derived cues. Using the derived cues to prompt the GenAI model can generate one of the multiple initial decisions. In some examples, using the meta-cues to prompt the GenAI model can also generate one of the initial decisions.

[0060] In various examples, the meta-hints and multiple derivative hints include instructions from the GenAI model for (i) analyzing a summary of the vulnerability report and a summary of the associated defect report set to classify the vulnerability report as valid or invalid, and (ii) generating a list of inference bugs that identifies one or more associated defect reports and classifies the vulnerability report as valid or invalid based on the one or more associated defect reports.

[0061] Then, at step 360, the method can generate a final decision on the validity of the vulnerability report based on multiple preliminary decisions.

[0062] In some examples, generating a final decision on the validity of a vulnerability report also includes calculating a weighted sum of multiple preliminary decisions.

[0063] In some examples, calculating the weighted sum involves determining the weights for multiple initial decisions. In various examples, the weights for the initial decisions can be determined based on how many relevant defect reports the GenAI model finds in the relevant defect report set to support the initial decision.

[0064] In some examples, the weighted sum falls within a range that is divided into three non-overlapping subranges. Based on which subrange the weighted sum falls into, the final decision may indicate (i) that the vulnerability report is valid, (ii) that the vulnerability report is invalid, or (iii) that the validity of the vulnerability report is uncertain.

[0065] Method 300 and any other methods described herein can be performed by computer-executable instructions (e.g., instructions that cause a computing system to perform the method) stored in one or more computer-readable media (e.g., storage devices or other tangible media) or stored in one or more computer-readable storage devices. Such methods can be performed in software, firmware, hardware, or a combination thereof. Such methods can be performed at least in part by a computing system (e.g., one or more computing devices).

[0066] The actions shown can be described from alternative perspectives while still implementing the techniques. For example, "sending" can also be described as "receiving" from a different perspective.

[0067] Example Overview of AI and Prompt Generation

[0068] Generative AI models, foundational models, and Large Language Models (LLMs) are interconnected concepts in the field of AI. The broad term generative AI encompasses AI systems that generate content such as text, images, music, or code. Unlike discriminative AI models, which aim to make decisions or predictions based on features of input data, generative AI models focus on creating new data points. Foundational models are a subset of these generative AI models, serving as a starting point for developing more specialized models. LLMs are a specific type of generative AI that works with language, understanding and generating human-like text. In the context of generative AI, including LLMs, cues are used as input or instructions to inform the AI ​​of desired content, context, or tasks. This allows users to guide the AI ​​to generate customized responses, interpretations, or creative content based on the provided cues.

[0069] In any of the examples in this article, LLMs can take the form of AI models designed to understand and generate human language. Such models typically leverage deep learning techniques, such as transformer-based architectures, to handle languages ​​with a very large number (e.g., billions) of parameters. Examples include Generative Pre-trained Transformer (GPT) developed by OpenAI, Google's Bidirectional Encoder Representation Transformer (BERT), a robustly optimized BERT pre-training method developed by Facebook AI, NVIDIA's Megatron-LM, and others. Pre-trained models are available from a variety of sources.

[0070] In any of the examples in this paper, cues can be provided to the LLM at runtime to generate responses. Cues in the LLM can be input instructions that guide the model's behavior. Cues can be textual hints, questions, or statements provided by the user to elicit a desired response from the LLM. Cues can act as primers in the model generation process. Cue sources can include user-generated queries, predefined templates, or system-generated suggestions. Technically, cues are tokenized and embedded into the model's input sequence as conditional signals for subsequent text generation. Techniques such as prefixes, temperature control, top-K sampling, and thought chains can be used to perform experiments with rapidly changing outputs to manipulate the results. These cues from different inputs and customized strategies allow users to influence the content generated by the LLM by shaping the underlying context and guiding the neural network's language generation. For example, cues can include instructions and / or examples to encourage the LLM to deliver results in a desired style and / or format.

[0071] Example architecture of LLM

[0072] Figure 4 An example architecture of the LLM 400 is shown, which can be used as the GenAI model described above (e.g., 180 or 280).

[0073] In the depicted example, the LLM 400 uses an autoregressive model (as implemented in OpenAI's GPT) to generate text content by predicting the next word in a sequence given previous words. Given its context, the LLM 400 can be trained to maximize the probability of each word in the training dataset.

[0074] like Figure 4 As shown, the LLM 400 can have an encoder 420 and a decoder 440, and their combination can be referred to as a "transformer". The encoder 420 processes the input text and transforms it into a context-rich representation. The decoder 440 takes this representation and generates the text output. In some examples, the LLM 400 may only include the decoder 440 (and no encoder 420).

[0075] For autoregressive text generation, the LLM 400 generates text sequentially, and for each word it generates, it depends on the context of the preceding words. During training, the model is learning to generate the target or output sequence presented to the decoder 440. However, compared to the output generated by the decoder 440 so far, the output is shifted one position to the right. In other words, the model sees the context of the preceding words and is responsible for predicting the next word. As a result, the LLM 400 can learn to generate text from left to right, which is how language is typically constructed.

[0076] The text input to encoder 420 can be preprocessed by input embedding unit 402. Specifically, input embedding unit 402 can segment the text input into a sequence of tokens, each token representing a word or a portion of a word. Each token can then be mapped to a fixed-length vector called the input embedding, which provides a continuous representation capturing the meaning and context of the text input. Similarly, to train LLM 400, the target or output sequence presented to decoder 440 can be preprocessed by output embedding unit 422. Similar to input embedding unit 402, output embedding unit 422 can provide a continuous representation or output embedding for each token in the output sequence.

[0077] Typically, the vocabulary in an LLM 400 is fixed and derived from the training data. The vocabulary in an LLM 400 consists of the lexical units generated during the training process. Words not in the vocabulary cannot be output. These lexical units are concatenated to form sentences in the text output.

[0078] In some examples, positional encoding (e.g., 404 and 424) can be performed to provide the order information of the lexical units generated by the input embedding unit 402 and the output embedding unit 422, respectively. Positional encoding is necessary because, unlike recurrent neural networks, the transformer processes all lexical units in parallel and does not inherently capture the order of lexical units. Without positional encoding, the model treats the sentence as a collection of words, thus losing the context provided by the word order. Positional encoding can be performed by mapping each position / index in the sequence to a unique vector and then adding it to the corresponding vector in the input or output embedding. By adding positional encoding to the input embedding, the model can understand the relative positions of words in the sentence. Similarly, by adding positional encoding to the output encoding, the model can preserve the word order when generating text output.

[0079] Each of the encoder 420 and decoder 440 may include multiple stacked or repeated layers (in Figure 4 (Nx represents the number of stacked layers in the encoder 420 and / or decoder 440). The number of stacked layers in the encoder 420 and / or decoder 440 can vary depending on the specific LLM architecture. Generally, a higher "N" usually means a deeper model, which can capture more complex patterns and dependencies in the data, but may require more computational resources for training and inference. In some examples, the number of stacked layers in the encoder 420 can be the same as the number of stacked layers in the decoder 440. In other examples, the LLM 400 can be configured so that the encoder 420 and decoder 440 can have different numbers of layers. For example, a deeper encoder (more layers) can be used to better capture the complexity of the input text, while a shallower decoder (fewer layers) can be used if the output generation task is less complex.

[0080] Encoder 420 and decoder 440 are related through a shared embedding and attention mechanism, which allows decoder 440 to access the contextual information generated by encoder 420, enabling LLM 400 to generate coherent and context-aware, accurate responses. In other words, the output of encoder 420 can be used as the basis for the decoder network to construct the generated text.

[0081] Both encoder 420 and decoder 440 include multi-layered attention and feedforward neural networks. The attention neural network implements an "attention" mechanism by calculating the relevance or importance of different words or lemmas within the input sequence to a given word or lemma in the output sequence, enabling the model to pay attention to context-related information while generating text. In other words, the attention neural network "attention" to certain parts of the sentence most relevant to the task of generating the text output. The feedforward neural network processes and transforms the information captured by the attention mechanism, applying non-linear transformations to the contextual embeddings of lemmas, enabling the model to learn complex relationships in the data and generate more context-accurate and expressive text.

[0082] exist Figure 4 In the example depicted, encoder 420 includes an internal attention or self-attention neural network 406 and a feedforward neural network 410, and decoder 440 includes a self-attention neural network 426 and a feedforward neural network 434. Self-attention neural networks 406 and 426 allow LLM 400 to weight the importance of different words or lexical units within the same input sequence (self-attention in encoder 420) and between the input and output sequences (self-attention in decoder 440), respectively.

[0083] Furthermore, decoder 440 also includes an attention-based encoder-decoder attention neural network 430, which receives input from the output of encoder 420. The encoder-decoder attention neural network 430 allows decoder 440 to focus on relevant portions of the input sequence (output of encoder 420) while generating the output sequence. As described below, the output of encoder 420 is a continuous representation or embedding of the input sequence. By feeding the output of encoder 420 into encoder-decoder attention neural network 430, contextual information and relationships captured in the input sequence (by encoder 420) can be carried over to decoder 440. This connection allows decoder 440 to access the entire input sequence, not just the final hidden state. Because decoder 440 can focus on all words in the input sequence, the input information can be aligned with the generation of the output to improve the contextual accuracy of the generated text output.

[0084] In some examples, one or more attention neural networks (e.g., 406, 426, 430) can be configured to implement a single-head attention mechanism, whereby the model captures relationships between words in an input sequence by assigning attention weights to each word based on its relevance to the target word. The term "single-head" indicates that only one set of attention weights or one mechanism exists to capture relationships between words in an input sequence. In some examples, one or more attention neural networks (e.g., 406, 426, 430) can be configured to implement a multi-head attention mechanism, whereby multiple sets of attention weights or "heads" capture different aspects of the input sequence in parallel. Each head learns different relationships and dependencies within the input sequence. These multiple attention heads can enhance the model's ability to focus on various features and patterns, enabling it to understand complex, multifaceted contexts, thus achieving more accurate and context-sensitive text generation. The outputs from multiple heads can be cascaded or linearly combined to produce the final attention output.

[0085] like Figure 4 As depicted, both encoder 420 and decoder 440 may include one or more addition and normalization layers (e.g., layers 408 and 412 in encoder 420, and layers 428, 432, and 436 in decoder 440). Addition layers, also known as residual connections, can add the output of another layer (e.g., an attention neural network or a feedforward network) to its input. After the addition operation, a normalization operation can be performed by a corresponding normalization layer, which normalizes the features (e.g., gives the features zero mean and unit variance), which can help stabilize the learning process and reduce training time.

[0086] The linear layer 442 at the output of the decoder 440 can transform the output embedding back into the original input space. Specifically, the output embedding generated by the decoder 440 is forwarded to the linear layer 442, which can transform the high-dimensional output embedding back into a space where each dimension corresponds to a word in the vocabulary of the LLM 400.

[0087] The output of the linear layer 442 can be fed into the softmax layer 444, which is configured to implement the softmax function, also known as softargmax or normalized exponential function, a generalization of a logistic function that compresses values ​​to a given range. Specifically, the softmax layer 444 takes the output (also known as the logarithm) from the linear layer 442 and converts them into probabilities. These probabilities sum to 1, and each probability corresponds to the likelihood that a particular word is the next word in the sequence. Typically, the word with the highest probability is chosen as the next word in the generated text output.

[0088] Still referencing Figure 4The following describes the general operation process by which the LLM 400 generates a response or text output in response to received prompt input.

[0089] First, the input text is segmented into a sequence of tokens, for example, by the input embedding unit 402, where each token represents a word or a portion of a word. Each token is then mapped to a fixed-length vector or input embedding. Position encoding 404 is then added to the input embedding to preserve information about the order of words in the input text.

[0090] Next, the input embedding is processed by the self-attention neural network 406 of the encoder 420 to generate a set of hidden states. As described above, a multi-head attention mechanism can be used to focus on different parts of the input sequence. The output from the self-attention neural network 406 is added to its input (residual connection) and then normalized at the addition and normalization layer 408.

[0091] The feedforward neural network 410 is then applied independently to each word. The feedforward neural network 410 includes fully connected layers with non-linear activation functions, allowing the model to capture complex interactions between words. The output from the feedforward neural network 410 is added to its input (residual connection) and then normalized at the addition and normalization layer 412.

[0092] Decoder 440 generates the next word in an autoregressive manner using the hidden state from encoder 420 and its own previous output sequence, such that sequential output is generated by focusing on previously generated words. Specifically, the output of encoder 420 (the input embedding processed by encoder 420) is fed into encoder-decoder attention neural network 430 of decoder 440, which allows decoder 440 to pay attention to all words in the input sequence. As described above, encoder-decoder attention neural network 430 can implement a multi-head attention mechanism, for example, by computing a weighted sum of all encoded input vectors, where the most relevant vector is assigned the highest weight.

[0093] The previous output sequence of decoder 440 is first segmented by output embedding unit 422 to generate an output embedding for each token in the output sequence. Similarly, positional embedding 424 is added to the output embedding to preserve information about the order of words in the output sequence.

[0094] The output embedding is processed by the self-attention neural network 426 of the decoder 440 to generate a set of hidden states. The self-attention mechanism allows each word in the text output to pay attention to all words in the input sequence as well as all previous words in the output sequence. The output from the self-attention neural network 426 is added to its input (residual connection) and then normalized at the addition and normalization layer 428.

[0095] The encoder-decoder attention neural network 430 receives the output embedding processed by the self-attention neural network 426 and the addition and normalization layers 428. Alternatively, the encoder-decoder attention neural network 430 also receives the output from the addition and normalization layer 412, which represents the input embedding processed by the encoder 420. By considering both the processed input embedding and the output embedding, the output of the encoder-decoder attention neural network 430 represents the output embedding considering both the input sequence and the previously generated output. As a result, the decoder 440 can generate an output sequence that is context-aligned with the input sequence.

[0096] The output from the encoder-decoder attention neural network 430 is added to a portion of its input (residual connection), namely the output from the addition and normalization layer 428, and then normalized at the addition and normalization layer 432. The normalized output from the addition and normalization layer 432 is then passed through the feedforward neural network 434. The output of the feedforward neural network 434 is then added to its input (residual connection), and then normalized at the addition and normalization layer 436.

[0097] The processed output embedding from decoder 440 passes through linear layer 442, which maps the high-dimensional output embedding back to the size of the vocabulary; that is, it transforms the output embedding into a space where each dimension corresponds to a word in the vocabulary. Softmax layer 444 then converts the output of linear layer 442 into probabilities, each probability corresponding to the likelihood that a particular word is the next word in the sequence. Finally, LLM 400 samples output lexical units from the probability distribution generated by softmax layer 444 (e.g., selecting the lexical unit with the highest probability), and these lexical units are added to the sequence of generated lexical units for the text output.

[0098] The above steps are repeated for each new lexical unit until the generated sequence ends or the maximum length is reached. Furthermore, if the encoder 420 and / or decoder 440 have multiple stacked layers, the steps performed by the encoder 420 and decoder 440 are repeated on each layer of the encoder 420 and decoder 440 to generate each new lexical unit.

[0099] Example vulnerability knowledge extraction

[0100] Typically, vulnerability reports generated by tools (e.g., BDBA, etc.) contain important information that can help determine whether a reported vulnerability is valid or invalid. This information may include the affected software and versions, potential impacts (e.g., code execution), and methods (of the identified vulnerable libraries) that can trigger or exploit the vulnerability. Due to the complex and structured nature of vulnerability reports, directly extracting this information can be challenging. As described in this article, vulnerability knowledge extractors (e.g., 130) can automatically extract vulnerability summaries using the GenAI model.

[0101] As an example, an AI agent (e.g., 250) can instruct a prompt generator (e.g., 240) to generate vulnerability knowledge extraction prompts based on the following prompt template:

[0102]

[0103]

[0104] This sample template serves as a structured guide for GenAI models generating vulnerability summaries. It includes placeholders within curly braces (e.g., {{NVD_DESCRIPTION}}, {{BDSA_SOLUTION}}, {{NVD_ID}}, etc.) to dynamically insert detailed information extracted from vulnerability reports. These placeholders allow GenAI models to customize content based on specific data fields, such as descriptions, technical details, solutions, version ranges, and identifiers (e.g., CVE and BDSA IDs). The template also specifies the required output format and key elements, ensuring a concise and comprehensive summary of the affected software, potential impact, triggering mechanisms, and mitigation strategies.

[0105] The generated vulnerability knowledge extraction hints can be used to populate these placeholders with relevant field information extracted from the vulnerability report (e.g., by parser 255). For example, the parser can generate objects including relevant fields extracted from the vulnerability report:

[0106]

[0107]

[0108] After using vulnerability knowledge extracted from the above-mentioned fields to provide hints to the GenAI model, the GenAI model can generate a structured and concise vulnerability summary, such as the following:

[0109]

[0110]

[0111] As mentioned above, the generated vulnerability summary can be further incorporated into subsequent decision-making prompts to generate preliminary decisions.

[0112] Overview of examples of expert knowledge extraction

[0113] Figure 5 The overall process of extracting expert knowledge related to the vulnerability database is illustrated in the diagram.500

[0114] As shown in the figure, the defect report database 570 may include a set of defect reports 502 or tickets previously collected from historical vulnerability reports. Each defect report 502 can identify the vulnerability libraries used by the historical software and the paths to the vulnerability libraries within the historical software. Defect reports 502 may also include other vulnerability information, such as symptoms, solutions, and how to reproduce the defect. In various examples, defect reports may also include discussion threads 506 or comments entered by the support team when team members analyze the vulnerability reports.

[0115] An expert knowledge extractor (e.g., 140) can automatically identify relevant defect report sets from the defect report database 170, as further described below. Once relevant defect reports are identified, the expert knowledge extractor can use a GenAI model to extract ticket summaries 542 or summaries of these defect reports, integrating key insights such as symptoms, solutions, reproduction steps, and related discussions or notes. As described above, the generated ticket summaries 542 can be further incorporated into subsequent decision prompts to generate initial decisions.

[0116] For example, an AI agent (e.g., 250) can instruct a hint generator (e.g., 240) to generate an expert knowledge-refined hint based on a hint template that includes a placeholder {bug_info}, which can be filled with specific details of the relevant defect report:

[0117]

[0118] Process 500 enables the system to base its analysis (and decisions) on relevant contexts and incorporate domain-specific knowledge (e.g., expert knowledge contributed by the support team), thereby improving the reliability of vulnerability assessments.

[0119] Example search of relevant defect reports

[0120] As mentioned above, the vulnerability report generated for the selected software can specify the vulnerable libraries used by the selected software and the paths to the vulnerable libraries within the selected software. Figure 6 An example process 600 is shown for retrieving a relevant set of defect reports (i.e., defect reports that are related to or have a similar context to the received vulnerability report) from a defect report database 670.

[0121] The defect report database 670 contains a set of defect reports 675 or tags previously generated from some historical software. Each defect report 675 may specify, for example, the vulnerable library (including the library's name and version) and the path to the vulnerable library used by the historical software in a brief description section (or a different field).

[0122] In the depicted example, the received vulnerability report specifies the vulnerable library through a unique vulnerability identifier (e.g., "Vulnerability ID" is "CVD-1111-2222"), the name and version of the vulnerable library (e.g., "Detected Library" is "xyz1.1.1.1"), and the full path of the vulnerable library within the selected software (e.g., "Detected Path" is "AAA / BBB / CCC / xyz1.1.1.1"). Such vulnerable library information 610 can be extracted by parsing the vulnerability report (e.g., via parser 255).

[0123] To identify a vulnerability report associated with a received vulnerability report, the vulnerability library information 610 extracted from the vulnerability report can be compared with the vulnerability library information contained in the vulnerability report 675 to find a contextual match, for example, based on the library name and path.

[0124] like Figure 6 As shown, process 600 can begin with preprocessing step 620, which removes version information from detected vulnerable libraries and their associated paths. For example, after preprocessing, the detected library path "AAA / BBB / CCC / xyz1.1.1.1" becomes "AAA / BBB / CCC / xyz", and the detected library "xyz1.1.1.1" becomes "xyz". The versionless paths and libraries 630 can be used as a basis for searching for related defect reports.

[0125] Using the preprocessed path and library 630, query 640 can be executed to identify defect report 675, whose brief description (or different fields) includes the detected path (e.g., “AAA / BBB / CCC / xyz”) and the detected library (e.g., “xyz”). In some examples, this query can be executed on defect report 675 stored in defect report database 670 using a string matching operation (e.g., SQL LIKE).

[0126] Defect reports that match fully detected paths and libraries are considered relevant defect reports, and the identifier of the defect report can be added to the result list 650 representing the relevant defect report set.

[0127] Result list 650 can be initialized as an empty set, and the count of relevant defect reports in this set can start from zero. When a relevant defect report is identified and added to result list 650, the count is incremented accordingly, as indicated by condition check 660, if the count of relevant defect reports in the set remains less than a predefined number (e.g., a threshold K). If the count reaches the predefined number, the process can stop searching and output the final list as relevant defect report set 690.

[0128] On the other hand, if, after evaluating all defect reports in database 670, the count of defect reports in result list 650 is determined to be less than a predefined number, the process can continue by iteratively comparing subpaths of the detected paths of the vulnerable library with the paths stored in the defect reports. In some examples, this iterative approach involves removing prefixes (ending with a path separator, such as " / ") from the detected paths 680 and performing a new search at each iteration. For example, if the initially detected path is "AAA / BBB / CCC / xyz", the first subpath would be "BBB / CCC / xyz", then "CCC / xyz", and so on. During each iteration, the query can be repeated to search for defect reports whose paths match the current subpath. For example, if a defect report in the database references "BBB / CCC / xyz" as a library path, it can be added to result list 650 during the second iteration. This iterative sub-path matching can continue until the result list 650 reaches a predefined number (e.g., K) or the maximum number of iterations is reached, as determined by the number of segments in the originally detected path (e.g., the number of path separators). This iterative process ensures that partial path matching can be considered even in the absence of an exact full path match.

[0129] The final set of 690 relevant defect reports can be sorted according to certain criteria. For example, defect reports matching the complete detected path may have higher priority and rank higher than defect reports matching shorter sub-paths. In the same iteration, defect reports identified based on the same complete path or sub-path can be further sorted in descending order by their creation time. This approach ensures that the most accurate matches are prioritized, while also favoring more recent and potentially more relevant defect reports when multiple matches exist.

[0130] Example Thinking Paradigm

[0131] Existing GenAI models may experience illusions during the decision-making process. To mitigate this risk and improve overall performance, auto-hint engineering techniques can be applied to dynamically refine meta-hints into multiple derived hints representing different thought paradigms. Each thought paradigm includes specific instructions on how to modify the GenAI model of the meta-hint. Some exemplary and non-exhaustive thought paradigms are listed below:

[0132]

[0133] Example Meta Hint

[0134] The following is a sample meta-hint, from which multiple derivative hints can be generated:

[0135]

[0136]

[0137] The above meta hint has four placeholders: {bugs_info}, {detected_lib}, {detected_path}, and {cve_info}. The {detected_lib} and {detected_path} placeholders can be filled with the name and path of the vulnerability library identified in the vulnerability report (e.g., ...). Figure 6 The vulnerability database information 610 contains the "detected libraries" and "detected paths". The {cve_info} placeholder can be filled with a vulnerability digest (e.g., 132 or 232) extracted from the vulnerability report (e.g., an exemplary vulnerability digest for CVE-0000-11111: [CVE ID] listed above). As mentioned above, the {bugs_info} placeholder can be filled with a ticket digest (e.g., 142 or 242) extracted from the relevant defect reports. For each relevant defect report, the ticket digest may include the following information: (1) a unique identifier for the defect report, (2) the validity status of the defect report (e.g., "valid" or "invalid") (e.g., determined by the support team), and (3) a summary of the defect report. The following are example ticket digests extracted from three relevant defect reports:

[0138]

[0139]

[0140] Example prompt template for generating derivative prompts

[0141] Below are example prompt templates that can be used to generate derived prompts ("refined_prompts") using different thinking paradigms, where the {PROMPT} placeholder receives the meta prompt and the {THINKING_STYLE} placeholder receives the corresponding thinking paradigm:

[0142]

[0143] Different thinking paradigms can be systematically applied to create a comprehensive set of derivative prompts, each instructing the GenAI model to approach the problem from a unique perspective. This effectively regulates the behavior of the GenAI model and reduces its sensitivity to erroneous or biased reasoning. For example, derivative prompts based on the critical analysis thinking paradigm can encourage the GenAI model to examine potential flaws in its reasoning, while those based on the sequential reasoning thinking paradigm instruct the GenAI model to ensure stepwise logical consistency in its conclusions. As another example, derivative prompts based on the reflective thinking paradigm enable the GenAI model to re-evaluate its output based on new evidence or different interpretations. These diverse prompts not only improve the quality of the generated responses but also help uncover loopholes that might be overlooked when relying on a single prompt.

[0144] Example Derivative Hints

[0145] In some examples, among many derived suggestions, a set of suggestions deemed to produce high performance can be selected for the adjudicator (e.g., 160 or 260) to subsequently generate multiple initial decisions. This set of derived suggestions can be selected, for example, during the training phase by evaluating the suggestions using a training dataset that includes multiple test vulnerability reports (samples) with known "valid" or "invalid" classifications. Each derived suggestion can be fed into the GenAI model to generate a decision on the validity of each test vulnerability report, and these decisions can be compared to known classifications to measure the classification accuracy of the GenAI model. Performance metrics (e.g., accuracy, precision, recall, F1 score, etc.) can be used to evaluate the validity of each derived suggestion, ensuring that the selected set produces reliable and accurate results.

[0146] As an example, below are four derivative tips (corresponding to four different thought paradigms) that were found to produce good performance on the training dataset (each includes the same placeholders as the meta tip: {bugs_info}, {detected_lib}, {detected_path}, and {cve_info}). Here is the first example derivative tip corresponding to the ninth thought paradigm listed above (let's think about it step by step):

[0147]

[0148]

[0149] The following is a second example of a derived tip corresponding to the fourth thinking paradigm (critical thinking) listed above:

[0150]

[0151]

[0152] The following is a third example of a derived tip corresponding to the tenth thinking paradigm listed above (plan step by step and implement with good concepts and explanations):

[0153]

[0154]

[0155] The following is a fourth example of a derived hint corresponding to the thirteenth thinking paradigm listed above (ignore the current solution...):

[0156]

[0157]

[0158] Example of initial decision

[0159] As described above, the AI ​​agent (e.g., 250) can individually prompt the GenAI model with multiple decision cues to generate multiple preliminary decisions regarding the validity of the vulnerability report. Each decision cue can be generated using selected derived cues or meta-cues as cue templates, populated with relevant contextual information such as the identified vulnerable libraries and their paths (e.g., {detected_lib} and {detected_path}), a vulnerability summary (e.g., {cve_info}), and a ticket summary (e.g., {bugs_info}), as described above.

[0160] For each decision prompt, the GenAI model can generate a response (preliminary decision) based on the indicated output format. As shown in the example above, the response generated by the GenAI model can be organized as a JSON object, which includes not only the binary classification of the vulnerability report (e.g., "Decision": "Valid" or "Invalid"), but also the root cause of such classification (e.g., "Cause"), a list of related vulnerability reports (or "References") that GenAI finds to support the classification (e.g., "Inferred Vulnerability"), and other information. For illustration, an exemplary response (preliminary decision) generated by the GenAI model using the aforementioned meta-prompt and four derived prompts is listed below.

[0161]

[0162]

[0163]

[0164] Example ruling

[0165] As described above, the arbitrator (e.g., 160 or 260) can use a majority voting mechanism to reach a final decision on the validity of a vulnerability report. In some examples, majority voting can be achieved by calculating a weighted sum of preliminary decisions. For example, for each decision cue (which may be based on a meta-cue or a derived cue), the GenAI model generates a binary classification (valid or invalid) along with supporting fundamentals and references to relevant vulnerability reports identified (inferred vulnerability). The weights assigned to the preliminary decisions can be proportional to the number of references linked to that decision. These weighted decisions can then be aggregated to generate a final decision. In a particular example, the final decision D can be calculated using the following formula. final :

[0166]

[0167] Here, N is the number of decision cues (e.g., the number of derived cues and meta-cues used to generate an initial decision), I i It is the number of inferred defects (i.e., the number of relevant defect reports identified for the i-th decision cue), D i K is the binary classification of the i-th decision prompt (e.g., 1 for valid, -1 for invalid), and K is the total number of relevant defect reports retrieved from the library based on the detected paths.

[0168] Based on the above formula, the number of inferred defects (I) in the relevant defect report set (total K) identified by the GenAI model as supporting this preliminary decision is used to determine the basis for the calculation. i To determine the weight of a particular preliminary decision.

[0169] Final Decision D final Normalization is performed in the range [-1, 1], which can be divided into three non-overlapping sub-ranges based on an upper threshold (U) and a lower threshold (L), which can be predefined values ​​between -1 and 1. If D final If the vulnerability report falls within the subrange between U and 1, it can be deemed valid, or if D final If the vulnerability report falls within the subrange between -1 and L, it can be deemed invalid. On the other hand, if D... finalIf the vulnerability report falls within the sub-range between L and U, its validity can be considered uncertain. Therefore, dividing the full range into three sub-ranges aggregates the initial binary classification of the decision into a final decision with three possible outcomes: valid, invalid, or uncertain. In a specific example, U can be set to 0, and L can be set to -0.5. Other thresholds can also be used.

[0170] Using the five preliminary decisions (N=5) listed in the previous section as an example, assume K=5 (i.e., a total of five relevant defect reports). Three preliminary decisions classify the defect reports as valid, while the other two classify them as invalid. Because each preliminary decision lists three inferred defects (“313774, 315333, 283252”), each preliminary decision has a weight of 3 / 5. Therefore, the final decision can be calculated as D. final =0.12. Therefore, the vulnerability report can be considered valid (assuming U=0 and L=-0.5).

[0171] As another example, suppose we use N=5 decision cues (e.g., based on four derived cues and one meta cue) and retrieve K=5 related defect reports (e.g., defect-1, defect-2, defect-3, defect-4, defect-5). Furthermore, assume the following preliminary decisions and weights are obtained: Decision 1 is "invalid" (D1=-1), weight I1 / K=2 / 5, because there are two inference flaws (e.g., flaw-2 and flaw-3); Decision 2 is "valid" (D2=1), weight I2 / K=3 / 5, because there are three inference flaws (e.g., flaw-1, flaw-2, and flaw-3); Decision 3 is "invalid" (D3=-1), weight I3 / K=4 / 5, because there are four inference flaws (e.g., flaw-1, flaw-3, flaw-4, and flaw-5); Decision 4 is "valid" (D4=1), weight I4 / K=5 / 5, because there are five inference flaws (e.g., flaw-1, flaw-2, flaw-3, flaw-4, and flaw-5); and Decision 5 is "valid" (D5=1), weight I5 / K=0 / 5, because no inference flaws were identified. Therefore, the final decision is calculated as D. final =0.08. Therefore, the vulnerability report can be considered valid (assuming U=0 and L=-0.5).

[0172] Example experimental results

[0173] To evaluate the automated software vulnerability assessment system disclosed in this paper, experiments were conducted using a dataset comprising 1,846 historical vulnerability reports (“samples”). These samples were matched with corresponding Bugzilla tickets, such as... Figure 7As summarized in Table 710. The dataset includes attributes of known vulnerabilities identified by their CVE identifiers. Of the total sample, 1,683 were classified as invalid, and only 163 were considered valid, indicating that approximately 91% of security Bugzilla tickets are unnecessary, leading to significant resource waste and inefficiency in vulnerability tracking.

[0174] The experimental evaluation uses the published method based on the normalized decision parameter D. final These vulnerability reports are categorized as valid, invalid, or uncertain. Figure 7 Table 720 presents the results using various numbers of decision hints N (e.g., N=1, 3, or 5). For those samples that could make valid / invalid decisions (excluding uncertain samples), precision and recall metrics were used to evaluate classification accuracy.

[0175] In Table 720, VV represents a valid input vulnerability that the method correctly evaluates as valid; VI represents a valid input vulnerability that the method incorrectly evaluates as invalid; IV represents an invalid input vulnerability that the method incorrectly evaluates as valid; and II represents an invalid input vulnerability that the method correctly evaluates as invalid. It is important to minimize VI to avoid rejecting valid vulnerabilities, which could lead to unresolved security risks. Conversely, it is important to maximize II, which ensures that invalid vulnerabilities are correctly identified, thereby reducing unnecessary effort and improving resource efficiency in processing vulnerability reports. In the illustrated example, the upper threshold U is set to 0, and the lower threshold L is set to -0.5 to balance risk (e.g., introducing VI) and efficiency improvement (e.g., maximizing II).

[0176] As shown in the figure, with N=5, the published method achieves 99.26% precision and 85.05% recall. In this configuration, the model evaluates 946 out of 1,846 Bugzilla tickets as invalid (including 939 correctly classified invalid tickets and 7 misclassified valid tickets), and therefore they are not opened for tracking. Thus, only 900 tickets (including 621 indeterminate tickets, 114 correctly classified valid tickets, and 165 misclassified invalid tickets) need to be opened, significantly reducing the number of invalid Bugzilla tickets.

[0177] Example Advantages

[0178] The technology described in this article offers several technical advantages, addressing key challenges in automated vulnerability verification and improving the efficiency of the software vulnerability management process.

[0179] One technological advantage lies in the integration of vulnerability knowledge extraction and expert knowledge extraction. By leveraging the analytical capabilities of GenAI and contextual information extracted from defect tracking systems, the disclosed solution combines human expertise (e.g., embedded in defect report databases) with automated analysis (e.g., extraction of vulnerability summaries and ticket summaries) to create a highly accurate and reliable vulnerability verification system. This hybrid approach mitigates the limitations of manual verification processes, which are often time-consuming and prone to human error, and ensures a more consistent assessment of software vulnerabilities.

[0180] Another technological advantage lies in the innovative use of automated suggestion engineering to improve decision-making accuracy. The disclosed system employs multiple suggestions with different thought paradigms to analyze vulnerability reports, enabling it to evaluate different perspectives on the same issue. This multi-suggestion approach is further enhanced by a weighted majority voting mechanism that aggregates preliminary decisions to arrive at a final decision on the validity of the vulnerability report. The adjudication of multiple preliminary decisions addresses the technical challenges of handling complex and ambiguous vulnerability reports, improving the robustness of the final vulnerability assessment.

[0181] Furthermore, the automation and real-time capabilities of the disclosed solution offer additional technical advantages. The disclosed system can process vulnerability reports in real time through integration with vulnerability detection tools such as BDBA. This real-time processing capability eliminates latency in vulnerability verification, enabling organizations to respond to security threats more quickly. Moreover, by automatically classifying valid, invalid, and uncertain vulnerabilities, the disclosed solution significantly reduces the manual workload associated with verifying a large number of vulnerability reports, thus scalable for enterprise applications.

[0182] In summary, the disclosed technology can improve the efficiency of software vulnerability assessment by combining GenAI-driven automation, real-time processing, and robust decision-making mechanisms, which simplifies the vulnerability verification workflow and optimizes resource utilization.

[0183] Example computing system

[0184] Figure 8 An example of a suitable computing system 800 in which the described innovation can be implemented is depicted. The computing system 800 is not intended to impose any limitation on the scope or functionality of this disclosure, as the innovation can be implemented in different computing systems.

[0185] refer to Figure 8 The computing system 800 includes one or more processing units 810, 815 and memories 820, 825. Figure 8In this document, the basic configuration 830 is included within the dashed lines. Processing units 810 and 815 can execute computer-executable instructions, such as those for implementing the features described in the examples herein (e.g., method 300). The processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multiprocessor system, multiple processing units can execute computer-executable instructions to increase processing power. For example, Figure 8 A central processing unit 810 and a graphics processing unit or coprocessor 815 are shown. Physical memories 820, 825 may be volatile memories (e.g., registers, caches, RAM), non-volatile memories (e.g., ROM, EEPROM, flash memory, etc.), or some combination thereof, accessible by the processing units(s) 810, 815. Memories 820, 825 may store one or more innovative software 880 implementing the invention in the form of computer-executable instructions suitable for execution by the processing units(s) 810, 815.

[0186] The computing system 800 may have additional features. For example, the computing system 800 may include a storage device 840, one or more input devices 850, one or more output devices 860, and one or more communication connections 870, including input devices, output devices, and communication connections for user interaction. Interconnection mechanisms (not shown), such as buses, controllers, or networks, may interconnect the components of the computing system 800. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system 800 and coordinates the activities of the components of the computing system 800.

[0187] The physical storage device 840 may be removable or non-removable and includes a magnetic disk, magnetic tape or tape cartridge, CD-ROM, DVD, or any other medium that can be used to store information in a non-transitory manner and is accessible within the computing system 800. The storage device 840 may store instructions for implementing one or more innovative software designs described herein.

[0188] Input device 850 may be an input device such as a keyboard, mouse, pen or trackball, a voice input device, a scanning device, a touch device (e.g., a touchpad, a display, etc.), or another device that provides input to computing system 800. Output device 860 may be a display, printer, speaker, CD burner, or another device that provides output from computing system 800.

[0189] The communication connection 870 enables communication to another computing entity via a communication medium. The communication medium can transmit information such as computer-executable instructions, audio or video input or output, or other data in modulated data signals. A modulated data signal is a signal whose characteristics are set or altered in a way that encodes information in the signal. By way of example and not limitation, the communication medium can be electrical, optical, RF, or other carriers.

[0190] Innovations can be described within the context of computer-executable instructions, such as those included in a program module, which execute on a computing system targeting a real or virtual processor (e.g., ultimately executing on one or more hardware processors). Typically, a program module or component can include routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The computer-executable instructions for a program module can execute within a local or distributed computing system.

[0191] For ease of description, specific embodiments use terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level descriptions of operations performed by a computer and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms vary depending on the implementation.

[0192] Computer-readable media

[0193] Any computer-readable medium described herein may be non-transitory (e.g., volatile memory such as DRAM or SRAM, non-volatile memory such as magnetic storage devices, optical storage devices, etc.) and / or tangible. Any storage operation described herein may be implemented by storage in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Anything described as being stored (e.g., data created and used during implementation) may be stored in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Computer-readable media may be limited to implementations that do not consist of signals.

[0194] Any method described herein may be implemented by computer-executable instructions in (e.g., stored thereon, encoded thereon, etc.) of one or more computer-readable media (e.g., computer-readable storage media or other tangible media) or one or more computer-readable storage devices (e.g., memory, magnetic storage device, optical storage device, etc.). Such instructions may cause a computing device to perform the method. The techniques described herein may be implemented in various programming languages.

[0195] Example cloud computing environment

[0196] Figure 9 An example cloud computing environment 900 in which the described technologies can be implemented is depicted, including, for example, systems 120, 220, and other systems described herein. The cloud computing environment 900 may include a cloud computing service 910. The cloud computing service 910 may include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. The cloud computing service 910 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0197] The cloud computing service 910 can be used by various types of computing devices (e.g., client computing devices), such as computing devices 920, 922, and 924. For example, computing devices (e.g., 920, 922, and 924) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablets or smartphones), or other types of computing devices. For example, computing devices (e.g., 920, 922, and 924) can utilize the cloud computing service 910 to perform computing operations (e.g., data processing, data storage, etc.).

[0198] In practice, it can support cloud-based, locally deployed, or hybrid scenarios.

[0199] Example Implementation

[0200] In any of the examples in this article, a software application (or “application”) may take the form of a single application or a suite of multiple applications, whether provided as a service (SaaS), in the cloud, on-premises, on desktop, on mobile devices, wearables, etc.

[0201] Although some of the operations of the disclosed methods are described in a specific order for ease of presentation, this description includes rearrangement unless the specific language used in this paper requires a particular order. For example, operations described in sequence may be rearranged or performed concurrently in some cases.

[0202] As described in this application and the claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly specifies otherwise. Furthermore, the term “comprising” means “including.” Additionally, “and / or” means “and” or “or,” as well as “and” and “or.”

[0203] Although specific prompt templates have been described above, it should be understood that these prompt templates are merely examples for illustrative purposes, and different prompt templates may be used based on the principles described herein.

[0204] In any of the examples described in this article, an operation performed at runtime or in real time means that the operation can be completed with negligible processing latency (e.g., the operation can be completed in 1 second, etc.).

[0205] Example Terms

[0206] Any of the following example terms can be implemented.

[0207] Clause 1. A computing system comprising: a memory; one or more hardware processors coupled to the memory; and one or more computer-readable storage media storing instructions, which, when loaded into the memory, cause the one or more hardware processors to perform operations including: receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report sets using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report using the generative AI model based on the summary of the vulnerability report and the summary of the relevant vulnerability report sets; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions.

[0208] Clause 2. The computing system according to Clause 1, wherein generating a summary of a vulnerability report includes: generating vulnerability knowledge extraction prompts based on a first prompt template, wherein the first prompt template includes instructions for a generative AI model to extract vulnerability information from a vulnerability library used by the selected software from the vulnerability report; and using the vulnerability knowledge extraction prompts to prompt the generative AI model.

[0209] Clause 3. The computing system according to any one of Clauses 1 to 2, wherein the retrieval comprises: searching a defect report database to identify a collection of defect reports for one or more historical software, wherein the defect reports of the historical software specify the vulnerability libraries used by the historical software and the paths of the vulnerability libraries within the historical software; and identifying a relevant set of defect reports from the collection of defect reports, wherein the identification comprises comparing the paths of the vulnerable libraries within the selected software with the paths of the vulnerability libraries within the historical software.

[0210] Clause 4. The computing system according to Clause 3, wherein the identification further includes: adding a vulnerability report of the historical software to a relevant vulnerability report set if (i) the path of the vulnerability library in the historical software contains the path of the vulnerable library in the selected software, and (ii) the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number; and in response to determining, after evaluating the set of vulnerability reports, that the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number, iteratively comparing the sub-path of the vulnerable library in the selected software with the path of the vulnerability library in the historical software, wherein the sub-path is derived by removing a prefix from the path of the vulnerable library in the selected software.

[0211] Clause 5. The computing system according to any one of Clauses 1 to 4, wherein generating a summary of the relevant defect report set includes: generating expert knowledge extraction prompts based on a second prompt template, wherein the second prompt template includes instructions for the generative AI model to extract a summary and validity classification of the relevant defect report set from the relevant defect report set; and using the expert knowledge extraction prompts to prompt the generative AI model.

[0212] Clause 6. A computing system pursuant to any one of Clauses 1 to 5, wherein generating multiple preliminary decisions regarding the validity of a vulnerability report comprises: obtaining multiple derived hints previously derived from a meta-hint using a generative AI model based on multiple predefined thought paradigms; and individually prompting the generative AI model using the multiple derived hints, wherein using the derived hints to prompt the generative AI model is configured to generate one of the multiple preliminary decisions.

[0213] Clause 7. The computational system of Clause 6, wherein the meta-hints and multiple derivative hints include instructions for a generative AI model to: (i) classify vulnerability reports as valid or invalid based on an analysis of a summary of a vulnerability report and a summary of a set of related defect reports, and (ii) generate an inferred defect list that identifies one or more related defect reports, the vulnerability reports being classified as valid or invalid based on one or more related defect reports.

[0214] Clause 8. The computing system as described in any one of Clauses 6 to 7, wherein generating a final decision on the validity of a vulnerability report further includes calculating a weighted sum of multiple preliminary decisions.

[0215] Clause 9. The computing system as described in Clause 8, wherein computing the weighted sum includes determining the weights of multiple preliminary decisions, wherein the weights of the preliminary decisions are determined based on how many relevant defect reports the generative AI model finds in the relevant defect report set to support the preliminary decisions.

[0216] Clause 10. A computing system pursuant to any one of Clauses 8 to 9, wherein the weighted sum is within a range divided into three non-overlapping sub-ranges, wherein, based on the sub-range into which the weighted sum falls, the final decision indicates (i) that the vulnerability report is valid, (ii) that the vulnerability report is invalid, or (iii) that the validity of the vulnerability report is uncertain.

[0217] Clause 11. A computer-implemented method comprising: receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report set using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report using the generative AI model based on the summary of the vulnerability report and the summary of the relevant vulnerability report set; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions.

[0218] Clause 12. The computer-implemented method according to Clause 11, wherein generating a summary of a vulnerability report includes: generating vulnerability knowledge extraction prompts based on a first prompt template, wherein the first prompt template includes instructions for a generative AI model to extract vulnerability information from a vulnerability library used by the selected software from the vulnerability report; and using the vulnerability knowledge extraction prompts to prompt the generative AI model.

[0219] Clause 13. The computer-implemented method according to any one of Clauses 11 to 12, wherein the retrieval comprises: searching a defect report database to identify a set of defect reports for one or more historical software, wherein the defect reports of the historical software specify the vulnerability libraries used by the historical software and the paths of the vulnerability libraries within the historical software; identifying a relevant set of defect reports from the set of defect reports, wherein the identification comprises comparing the paths of the vulnerable libraries within the selected software with the paths of the vulnerability libraries within the historical software.

[0220] Clause 14. The computer-implemented method according to Clause 13, wherein the identification further comprises: adding a vulnerability report of the historical software to a relevant vulnerability report set if (i) the path of a vulnerability database within the historical software contains a path of a vulnerable library within the selected software, and (ii) the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number; and in response to determining, after evaluating the set of vulnerability reports, that the count of vulnerability reports in the relevant vulnerability report set is less than a predefined number, iteratively comparing a subpath of a vulnerable library within the selected software with a path of a vulnerability database within the historical software, wherein the subpath is derived by removing a prefix from the path of a vulnerable library within the selected software.

[0221] Clause 15. The computer-implemented method according to any one of Clauses 11 to 14, wherein generating a summary of the relevant defect report set comprises: generating expert knowledge extraction prompts based on a second prompt template, wherein the second prompt template includes instructions for the generative AI model to extract a summary and validity classification of the relevant defect report set from the relevant defect report set; and using the expert knowledge extraction prompts to prompt the generative AI model.

[0222] Clause 16. A computer-implemented method according to any one of Clauses 11 to 15, wherein generating multiple preliminary decisions regarding the validity of a vulnerability report comprises: obtaining multiple derived hints previously derived from a meta-hint using a generative AI model based on multiple predefined thought paradigms; and individually prompting the generative AI model using the multiple derived hints, wherein prompting the generative AI model using the derived hints is configured to generate one of the multiple preliminary decisions.

[0223] Clause 17. The computer implementation method according to Clause 16, wherein the meta-hints and multiple derivative hints include instructions for a generative AI model to: (i) classify vulnerability reports as valid or invalid based on analysis of summaries of vulnerability reports and summaries of related defect report sets, and (ii) generate an inferred defect list that identifies one or more related defect reports, the vulnerability reports being classified as valid or invalid based on one or more related defect reports.

[0224] Clause 18. The computer-implemented method according to any one of Clauses 16 to 17, wherein generating a final decision on the validity of a vulnerability report further includes calculating a weighted sum of multiple preliminary decisions.

[0225] Clause 19. The computer-implemented method according to Clause 18, wherein calculating the weighted sum includes determining weights for a plurality of preliminary decisions, wherein the weights of the preliminary decisions are determined based on how many relevant defect reports the generative AI model finds in the relevant defect report set to support the preliminary decisions.

[0226] Clause 20. One or more non-transitory computer-readable media having computer-executable instructions encoded thereon, the computer-executable instructions causing one or more processors to perform a method comprising: receiving a vulnerability report for selected software, wherein the vulnerability report specifies vulnerable libraries used by the selected software and paths of the vulnerable libraries within the selected software; generating a summary of the vulnerability report using a generative artificial intelligence (AI) model; retrieving a set of relevant vulnerability reports for the specified vulnerable libraries from a vulnerability report database; generating a summary of the relevant vulnerability report set using the generative AI model; generating multiple preliminary decisions regarding the validity of the vulnerability report using the generative AI model based on the summary of the vulnerability report and the summary of the relevant vulnerability report set; and generating a final decision regarding the validity of the vulnerability report based on the multiple preliminary decisions.

[0227] Technology from any of these terms may be combined with technology described in any one or more other terms.

[0228] Given the many possible embodiments in which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. Rather, the scope of the disclosed technology includes what is covered by the scope and spirit of the appended claims.

Claims

1. A computing system, comprising: Memory; One or more hardware processors coupled to the memory; and One or more computer-readable storage media storing instructions that, when loaded into memory, cause the one or more hardware processors to perform operations, the operations including: Receive vulnerability reports for the selected software, wherein the vulnerability reports specify the vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; Generate a summary of the vulnerability report using a generative artificial intelligence (AI) model; Retrieve a set of relevant defect reports for the specified vulnerable library from the defect report database; Generative AI models are used to generate summaries for relevant defect report sets. Based on summaries of vulnerability reports and summaries of related defect report sets, a generative AI model is used to generate multiple preliminary decisions regarding the validity of vulnerability reports; and A final decision regarding the validity of the vulnerability report is generated based on the aforementioned preliminary decisions.

2. The computing system of claim 1, wherein, The summary of the generated vulnerability report includes: Based on the first prompt template, vulnerability knowledge extraction prompts are generated, wherein the first prompt template includes instructions for the generative AI model to extract vulnerability information from the vulnerability database used by the selected software from the vulnerability report; and Leveraging vulnerability knowledge to extract hints for generative AI models.

3. The computing system of claim 1, wherein, The search includes: Search the defect report database to identify a collection of defect reports for one or more historical software programs, where the defect reports specify the vulnerability libraries used by the historical software and the paths to those libraries within the historical software; and Identify relevant vulnerability report sets from the collection of vulnerability reports, wherein identification includes comparing the paths of vulnerable libraries within the selected software with the paths of vulnerability libraries within historical software.

4. The computing system of claim 3, wherein, The identification also includes: If (i) the path to the vulnerability database within the historical software contains the path to the vulnerable database within the selected software, and (ii) the count of defect reports in the relevant defect report set is less than a predefined number, then add the defect reports of the historical software to the relevant defect report set; and In response to determining, after evaluating the set of defect reports, that the count of defect reports in the relevant defect report set is less than a predefined number, the subpath of the selected vulnerable library within the software is iteratively compared with the path of the vulnerability library within the historical software, wherein the subpath is derived by removing prefixes from the path of the selected vulnerable library within the software.

5. The computing system of claim 1, wherein, A summary of the generated defect report set includes: Expert knowledge extraction prompts are generated based on the second prompt template, which includes instructions for the generative AI model to extract summaries and validity classifications of relevant defect report sets; and Use expert knowledge to extract hints to prompt generative AI models.

6. The computing system of claim 1, wherein, Several initial decisions regarding the effectiveness of vulnerability reports include: Based on multiple predefined thinking paradigms, a generative AI model is used to obtain multiple derived hints previously derived from meta-hints; and The generative AI model is individually prompted using the plurality of derived prompts, wherein the generative AI model is configured to generate one of the plurality of preliminary decisions using derived prompts.

7. The computing system of claim 6, wherein, The meta-hints and the plurality of derived hints include instructions for a generative AI model to: (i) classify vulnerability reports as valid or invalid based on analysis of summaries of vulnerability reports and summaries of related defect report sets, and (ii) generate an inferred defect list that identifies one or more related defect reports, based on which the vulnerability reports are classified as valid or invalid.

8. The computing system of claim 6, wherein, The final decision on the validity of the vulnerability report also includes calculating a weighted sum of the multiple preliminary decisions.

9. The computing system of claim 8, wherein, The weighted sum calculation includes determining the weights of the plurality of preliminary decisions, wherein the weights of the preliminary decisions are determined based on how many relevant defect reports the generative AI model finds in the relevant defect report set to support the preliminary decisions.

10. The computing system of claim 8, wherein, The weighted sum falls within three non-overlapping sub-ranges, where, based on the sub-range into which the weighted sum falls, the final decision indicates (i) the vulnerability report is valid, (ii) the vulnerability report is invalid, or (iii) the validity of the vulnerability report is uncertain.

11. A computer-implemented method, comprising: Receive vulnerability reports for the selected software, wherein the vulnerability reports specify the vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; Generate a summary of the vulnerability report using a generative artificial intelligence (AI) model; Retrieve a set of relevant defect reports for the specified vulnerable library from the defect report database; Generative AI models are used to generate summaries for relevant defect report sets. Based on summaries of vulnerability reports and summaries of related defect report sets, a generative AI model is used to generate multiple preliminary decisions regarding the validity of vulnerability reports; and A final decision regarding the validity of the vulnerability report is generated based on the aforementioned preliminary decisions.

12. The computer-implemented method of claim 11, wherein, The summary of the generated vulnerability report includes: Based on the first prompt template, vulnerability knowledge extraction prompts are generated, wherein the first prompt template includes instructions for the generative AI model to extract vulnerability information from the vulnerability database used by the selected software from the vulnerability report; and Leveraging vulnerability knowledge to extract hints for generative AI models.

13. The computer-implemented method according to claim 11, wherein, The search includes: Search the defect report database to identify a collection of defect reports for one or more historical software programs, where the defect reports specify the vulnerability libraries used by the historical software and the paths to the vulnerability libraries within the historical software. as well as Identify relevant vulnerability report sets from the collection of vulnerability reports, wherein identification includes comparing the paths of vulnerable libraries within the selected software with the paths of vulnerability libraries within historical software.

14. The computer-implemented method according to claim 13, wherein, The identification also includes: If (i) the path to the vulnerability database within the historical software contains the path to the vulnerable database within the selected software, and (ii) the count of defect reports in the relevant defect report set is less than a predefined number, then add the defect reports of the historical software to the relevant defect report set; and In response to determining, after evaluating the set of defect reports, that the count of defect reports in the relevant defect report set is less than a predefined number, the subpath of the selected vulnerable library within the software is iteratively compared with the path of the vulnerability library within the historical software, wherein the subpath is derived by removing prefixes from the path of the selected vulnerable library within the software.

15. The computer-implemented method according to claim 11, wherein, A summary of the generated defect report set includes: Expert knowledge extraction prompts are generated based on the second prompt template, which includes instructions for the generative AI model to extract summaries and validity classifications of relevant defect report sets; and Use expert knowledge to extract hints to prompt generative AI models.

16. The computer-implemented method of claim 11, wherein, Several initial decisions regarding the effectiveness of vulnerability reports include: Based on multiple predefined thinking paradigms, a generative AI model is used to obtain multiple derived hints previously derived from meta-hints; and The generative AI model is individually prompted using the plurality of derived prompts, wherein the generative AI model is configured to generate one of the plurality of preliminary decisions using derived prompts.

17. The computer-implemented method of claim 16, wherein, The meta-hints and the plurality of derived hints include instructions for a generative AI model to: (i) classify vulnerability reports as valid or invalid based on analysis of summaries of vulnerability reports and summaries of related defect report sets, and (ii) generate an inferred defect list that identifies one or more related defect reports, based on which the vulnerability reports are classified as valid or invalid.

18. The computer-implemented method of claim 16, wherein, The final decision on the validity of the vulnerability report also includes calculating a weighted sum of the multiple preliminary decisions.

19. The computer-implemented method of claim 18, wherein, The weighted sum calculation includes determining the weights of the plurality of preliminary decisions, wherein the weights of the preliminary decisions are determined based on how many relevant defect reports the generative AI model finds in the relevant defect report set to support the preliminary decisions.

20. One or more non-transitory computer-readable media having computer-executable instructions encoded thereon, the computer-executable instructions causing one or more processors to perform a method, the method comprising: Receive vulnerability reports for the selected software, wherein the vulnerability reports specify the vulnerable libraries used by the selected software and the paths of the vulnerable libraries within the selected software; Generate a summary of the vulnerability report using a generative artificial intelligence (AI) model; Retrieve a set of relevant defect reports for the specified vulnerable library from the defect report database; Generative AI models are used to generate summaries for relevant defect report sets. Based on summaries of vulnerability reports and summaries of related defect report sets, a generative AI model is used to generate multiple preliminary decisions regarding the validity of vulnerability reports; and A final decision regarding the validity of the vulnerability report is generated based on the aforementioned preliminary decisions.

Images