Deep learning framework fuzzy testing method based on large model cue word optimization

Abstract

The invention relates to the cross technical field of artificial intelligence and software security testing, in particular to a deep learning framework fuzz testing method based on large model cue word optimization, which is used for improving vulnerability mining efficiency and testing intelligence level of fuzz testing on a deep learning framework. According to the method, the advantages of a large language model in the aspects of code understanding and generation are fully utilized, and efficient vulnerability detection of a deep learning framework is realized by introducing a cue word adaptive optimization and variation mechanism. The method mainly comprises the following steps: (1) providing a deep learning framework API classification method and a cue word routing mechanism; (2) proposing a large model cue word adaptive optimization mechanism; and (3) proposing a deep learning framework fuzzy test variation strategy and a dynamic selection mechanism. According to the method, the automation and vulnerability discovery capability of fuzzy testing can be remarkably improved while the generation quality is ensured, and the method has relatively high universality and application value.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and software security testing, specifically to a fuzzing method for a deep learning framework based on large model prompt word optimization, which is used to improve the vulnerability discovery efficiency and testing intelligence level of fuzzing on a deep learning framework. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI has deeply integrated into all aspects of human production and life, profoundly impacting socio-economic structures, industrial forms, and even daily lifestyles. Whether it's intelligent voice assistants, autonomous driving, or medical image diagnosis and intelligent recommendation systems, these widespread applications rely heavily on the powerful capabilities of deep learning technology. Deep learning frameworks (such as TensorFlow, PyTorch, and MindSpore) play a crucial role in the entire AI ecosystem as the core foundational software supporting the development and operation of AI application models. Specifically, they are responsible for model construction, training, and inference; their security and stability directly affect the reliable operation of upper-layer AI applications. If a framework itself has security vulnerabilities or stability defects, it may not only lead to abnormal model behavior but also trigger systemic security risks. Therefore, conducting fuzz testing on deep learning frameworks to identify potential vulnerabilities and security risks has become a crucial research direction in the current cybersecurity field.

[0003] Currently, fuzzing methods for deep learning frameworks can be broadly categorized into two types: model-level fuzzing and API-level fuzzing. Model-level testing typically generates complete neural network structures for detection, but it struggles to pinpoint vulnerabilities at the API level. API-level fuzzing, on the other hand, focuses on specific interfaces within the framework. It automatically or semi-automatically generates API call code as test cases, typically achieving broader API coverage compared to model-level fuzzing. In recent years, researchers have proposed various API-level testing techniques. For example, the paper "Deng, Yinlin, et al. Fuzzing deep-learning libraries via automated relational apiinference." Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2022" significantly improves API-level fuzzing coverage by automatically inferring relationships between APIs and performing differential testing. However, this method still has limitations in handling complex API combinations and diverse input semantics, making it difficult to cover deep interaction logic. The paper "Xie, Danning, et al. Docter: Documentation-guided fuzzing for testing deep learning API functions. Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis. 2022" extracts constraint information from API documentation to guide input generation. However, inconsistent documentation descriptions and semantic ambiguities often lead to low generation efficiency and a high proportion of invalid samples.

[0004] With the rise of Large Language Model (LLM) technology, researchers have noticed its advantages in training on massive programming corpora, enabling it to understand code logic and generate high-quality program snippets. Consequently, an increasing number of studies are exploring the use of large models to generate test cases, aiming to improve the automation and intelligence of fuzzing in deep learning frameworks. For example, the paper "Deng, Yinlin, et al. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models." (Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis. 2023) investigated how to directly generate code samples for testing deep learning libraries using large language models. While this method demonstrates the potential of LLM in test case generation, its prompt design remains relatively fixed, lacking adaptive optimization to API semantic characteristics and model preferences. Furthermore, the generated samples exhibit strong homogeneity in structure and logic, leading to insufficient coverage of vulnerability triggering paths.

[0005] In summary, current fuzzing methods based on LLM (Large Language Model) for deep learning frameworks suffer from the following problems: First, in terms of cue word design, existing methods fail to adequately balance the characteristics of deep learning framework APIs with testing efficiency, resulting in low efficiency in test case generation and increasing the cost of calling and generating large models. Second, existing methods typically rely on fixed, templated cue words, neglecting the sensitivity of different large language models to cue word styles and semantic structures, thus affecting the generation performance and reliability of large models in fuzzing. Third, large language models suffer from insufficient diversity in test case generation; the generated samples tend to be homogeneous in semantics and structure, lacking sufficient coverage of potential vulnerability triggering paths and boundary conditions, thereby limiting their vulnerability discovery capabilities. Therefore, there is an urgent need for a method to optimize the cue word design of large models, improving model performance while considering computational overhead, thereby enhancing the overall effectiveness of fuzzing within deep learning frameworks.

[0006] To address the aforementioned challenges, this invention proposes a deep learning framework fuzzing method based on large model prompt word optimization. This method fully leverages the inherent advantages of large language models in code understanding and generation, introducing an innovative prompt word adaptive optimization and mutation mechanism to achieve efficient vulnerability discovery within deep learning frameworks. First, based on the characteristics of deep learning framework APIs, APIs are categorized into three types according to functional complexity, API documentation completeness, and the knowledge coverage and timeliness of the large model. Corresponding basic prompt word templates are designed for each category, and a prompt word routing mechanism is constructed to improve the matching degree between prompt words and API characteristics. Second, to achieve adaptive matching between prompt strategies and model characteristics, this invention combines the selected large language model and employs adaptive prompt word optimization technology to dynamically fine-tune the three types of templates. This ensures that testing efficiency is effectively improved while maintaining generation quality. Finally, to enhance the diversity of test cases and vulnerability disclosure capabilities, this invention designs four types of prompt word mutation strategies, including input data mutation, parameter combination mutation, API combination mutation, and exploratory mutation based on a historical vulnerability database. Meanwhile, a multi-armed slot machine algorithm is introduced to dynamically select mutation strategies and update weights, thereby improving the diversity and boundary coverage of samples generated by large models, and effectively enhancing the vulnerability discovery capability of deep learning framework fuzzing. In summary, the method proposed in this invention has good application prospects and promotional value. Summary of the Invention

[0007] This invention proposes a fuzz testing method for deep learning frameworks based on large-model cue word optimization. It aims to address the shortcomings of existing methods in cue word design, generation performance, and test case diversity, thereby effectively improving the vulnerability discovery capabilities of deep learning frameworks. The main contents include:

[0008] (1) A deep learning framework API classification method and prompt word routing mechanism are proposed. APIs are classified according to their characteristics (functional complexity, documentation completeness, knowledge coverage of large models, and timeliness), and corresponding prompt word routing strategies are constructed to achieve efficient matching between prompt words and API characteristics.

[0009] (2) A large model prompt word adaptive optimization mechanism is proposed. For the basic prompt word templates of different API categories, adaptive optimization technology is used to dynamically optimize the model in combination with specific LLMs to ensure a balance between generation quality and testing efficiency.

[0010] (3) A deep learning framework fuzzing mutation strategy and dynamic selection mechanism are proposed. Four types of prompt word mutation strategies (input data, parameter combination, API combination, and historical vulnerability database exploration) are introduced, and dynamic selection and weight update are performed in combination with the multi-armed slot machine algorithm to enhance the diversity of test cases, boundary coverage and vulnerability reach.

[0011] The details are as follows:

[0012] (1) The performance of large models in fuzzing within deep learning frameworks (including generation speed, cost, and generation effectiveness) is highly dependent on the design of the prompt words. However, existing methods often neglect efficiency in prompt word design, using a fixed prompt word template for all APIs. These fixed prompt words typically embed a large amount of documentation information and few-shot examples, resulting in verbose prompt words. This is especially problematic in deep learning frameworks where the number of APIs is large and their complexity varies significantly, leading to resource waste and low fuzzing efficiency. An intuitive approach is to use simplified prompt words to improve generation efficiency for common or simple APIs, while using more complete and information-rich prompt words to improve generation effectiveness for complex APIs. To address the diversity and complexity of numerous APIs in deep learning frameworks, this invention proposes a quantitative evaluation and grading mechanism for classifying APIs and constructing prompt word routing strategies. First, the functional complexity and documentation completeness of the APIs are quantified as classification criteria. Functional complexity F is calculated using the number of API parameters P and the number of dependencies between parameters Q, reflecting the complexity of calls and logic. Therefore, the functional complexity index F is calculated as follows:

[0013]

[0014] The more parameters there are and the more complex the parameter dependencies, the higher the functional complexity, i.e., an F-value close to 1 indicates functional complexity. Secondly, the API documentation completeness D is quantified. Large models may have already learned the API documentation information during training, thus avoiding repeated embedding in the prompts and saving prompt length. Mainstream framework API documentation typically includes five categories: API signature, API function description, API parameter description, API return type, and API code example. Therefore, the documentation completeness D is calculated as follows:

[0015]

[0016] Where d represents the number of the five types of information (API signature, function description, parameter description, return type, and sample code) present in the document. Finally, the knowledge coverage and timeliness K of the large model are quantified, considering two scenarios: First, due to the complexity of the API, the large model may be unable to generate effective test cases, reflecting limitations in its understanding and reasoning abilities. Second, the API may be newly released or have outdated knowledge, meaning the large model's training data does not include the API, thus limiting its generation capabilities. Therefore, for each API, the target large language model attempts to generate 5 test cases, and K is calculated based on the number of effective test case generations.

[0017]

[0018] An overly complex API or outdated model knowledge will result in a smaller K. Finally, after normalizing the functional complexity F, documentation completeness D, and knowledge coverage K, a comprehensive scoring formula is used to calculate the total score S for each API:

[0019]

[0020] Based on the overall score S, APIs are divided into three categories: Category A (simple API), S≥0.7; Category B (medium API), 0.4≤S<0.7; and Category C (complex API), S<0.4.

[0021] This invention addresses the three types of APIs (simple APIs, medium APIs, and complex APIs) mentioned above. Based on their comprehensive evaluation score S, corresponding basic prompt word templates are designed, and a prompt word routing mechanism is constructed to ensure efficient matching between the prompt strategy and API characteristics. First, for simple APIs, we adopt direct instruction-type prompt words (Instruction, TargetAPI). This template has a concise structure, containing only a clear task instruction (Instruction, e.g., "Generate a Python test case that calls this API") and the name or signature of the target API (TargetAPI). It aims to guide large models to generate minimal runnable test cases or perform mutation tasks in the most concise and efficient way. Its advantages include fast generation speed, low computational cost, and suitability for APIs with simple functions and clear parameter structures. Large models can accurately understand and generate effective test cases without requiring extensive context. Second, for medium APIs, this invention designs enhanced prompt words (Role, Instruction, TargetAPI, APIContext). This template, building upon direct instruction prompts, introduces role settings (e.g., "You are a professional deep learning framework test engineer") and rich contextual information (APIContext), such as API function descriptions, parameter types, value ranges, and constraints. This assists large models in generating test cases with more reasonable context, more accurate parameter configurations, and compliance with API semantic specifications, thereby improving testing effectiveness. Finally, to address the challenges posed by complex APIs, this invention proposes chain-of-thought (CoT) guided prompts. Building upon enhanced prompts, this template guides large models through a detailed chain-of-thought (CoT) mechanism, leading to step-by-step reasoning. The reasoning paradigm may include step-by-step reasoning instructions (e.g., "First analyze the API function, then consider its parameter combinations, then design abnormal inputs, and finally generate code"), complex input / output examples, and potential vulnerability pattern hints. The aim is to guide large models to deeply understand the internal mechanisms, interaction logic, and potential boundary conditions of complex APIs, generating test cases for complex boundaries, multi-API joint calls, abnormal inputs, or specific vulnerability triggering paths.

[0022] (2) This invention proposes an adaptive optimization mechanism for prompt words in large-scale models, aiming to address the common problem in existing methods that neglect the sensitivity of different large-scale language models to prompt word style and semantic structure, thereby effectively improving testing efficiency while ensuring generation quality. The core of this mechanism lies in dynamically optimizing three basic prompt word templates: direct instruction, enhanced, and thought chain guided, to achieve optimal matching with the characteristics of the selected specific large-scale model. To achieve the above objectives, this invention first constructs a quantifiable comprehensive performance evaluation index system to accurately measure the optimization effect of prompt words. Specifically, the statistical indicators include generation effectiveness, API call relevance, code coverage, generation efficiency, and computational cost. Generation effectiveness is the proportion of test case code generated by the large-scale model that can be successfully executed without runtime errors (such as Python interpreter errors, framework API call exceptions, etc.). This indicator is quantified as... Where N valid N represents the number of valid test cases. total The total number of test cases generated. API call relevance checks whether the code generated by the large model contains the target API, thus effectively avoiding the illusion that the large model might create. This metric is quantified as... Where N relevant This refers to the number of valid test cases that contain correct API calls. Code coverage is the percentage of test cases generated by a large statistical model that cover the code path of the deep learning framework; this metric is quantified as... Code cov It is the number of lines of code covered by the test cases. total This refers to the total number of lines of code in the deep learning framework. Generation efficiency is the time cost required for the large model to generate complete test case code from receiving prompts. Finally, there's computational cost, which is the number of tokens consumed by the large model during test case generation (the sum of the input prompt tokens and the output response tokens). These metrics collectively constitute the evaluation criteria for the large model's prompt-adaptive optimization mechanism and serve as a precise measure of individual fitness in subsequent evolutionary algorithms.

[0023] Building upon this evaluation framework, this invention introduces an adaptive optimization method based on Evolutionary Algorithms (EA) to dynamically tune three basic prompt word templates and their variants. Notably, this optimization process does not cover all APIs; instead, it randomly selects a portion of each of the simple, medium, and complex API categories from the corresponding model as a training set, and optimizes each API category independently. This strategy ensures the efficiency and generalization of the optimization process. In this evolutionary algorithm, each prompt word to be optimized is abstracted as an "individual," whose core structure consists of variable components such as instructions, role settings, API context, thought chains, and few-shot examples. The algorithm first initializes the basic template prompt words and, under the constraint of semantic equivalence, drives the large model to generate an initial population of candidate prompt words. Subsequently, the "fitness" of each "individual" is precisely quantified according to the aforementioned evaluation criteria, specifically obtained by running LLM-generated test cases on randomly sampled APIs and evaluating their performance. To systematically explore the potential impact of different prompt word styles on the performance of large models and to serve as a key mutation mechanism in the iterative process of evolutionary algorithms, this invention meticulously designs a series of fine-grained prompt word variant generation strategies. These strategies are mainly reflected in the following aspects: at the semantic level, operations such as synonym replacement, sentence reconstruction, and voice conversion are performed on key elements such as instruction, APIContext, and CoT to generate prompt words with the same semantic function but diverse expressions; at the structural level, the optimal structural layout is explored by changing the order of the components within the prompt word, adding or removing guiding statements, and flexibly adjusting the number and presentation format of few-shot examples; at the tone and style level, the overall tone and style of the prompt words are adjusted to better adapt to the preferences of specific large-scale language models. In addition, for prompt word templates involving few-shot examples, we have also introduced an example selection and sorting strategy to dynamically select examples of different complexities and optimize their arrangement order in the prompt words, further improving the prompting effect.

[0024] (3) This invention proposes a fuzzing mutation strategy and dynamic selection mechanism for deep learning frameworks to address the issues of insufficient test case diversity and vulnerability discovery capabilities in existing work. This mechanism introduces four carefully designed cue word mutation strategies and combines them with a Multi-Armed Bandit (MAB) algorithm for dynamic selection and weight updates, significantly enhancing the diversity, boundary coverage, and vulnerability reach of test cases. Specifically, this invention designs the following four core mutation strategies, each targeting potential vulnerabilities at different levels within the deep learning framework. First, input data mutation focuses on diversifying the input tensors in large model-generated test cases to explore their impact on the model's internal processing logic. This strategy covers geometric transformations (such as random rotation, flipping, pruning, and scaling), numerical perturbations (such as adding Gaussian noise, numerical range compression / expansion, and data type replacement), and structural adjustments (such as changing tensor dimensions and input parameter shapes), aiming to trigger potential problems in input validation, memory access, data flow processing, and numerical stability. Second, parameter combination mutation focuses on exploring the parameter space of deep learning APIs, particularly the complex interactions between parameters. This strategy goes beyond independent mutation of a single parameter, emphasizing "multi-parameter collaborative mutation." This involves simultaneously selecting multiple related API parameters and mutating them within their respective supported valid non-boundary value ranges to discover errors caused by improper parameter combinations. It also considers different calling methods or parameter passing methods for the same API to detect consistency within the framework's internal implementation. Third, API combination mutation focuses on structural mutations in API call sequences within test cases to discover dependencies between APIs, state management issues, or improper call sequences. Specific operations include inserting or deleting APIs from existing sequences, replacing them with semantically similar but potentially different implementation details, and rearranging API call order while maintaining semantic rationality. This aims to discover API dependencies, state management issues, resource leaks, and runtime errors caused by improper API sequences. Finally, the historical vulnerability database exploration strategy utilizes known historical vulnerability information to guide fuzzing towards high-risk areas. By constructing a historical vulnerability database for the deep learning framework, each vulnerability is analyzed in depth to extract its root cause and triggering pattern. During fuzzing, a vulnerability pattern is randomly selected, and its corresponding mutation rules or test scenarios are migrated to the API currently under test, thereby efficiently rediscovering known vulnerability types and their variants.

[0025] To efficiently and dynamically select among the four mutation strategies mentioned above, this invention introduces the Multi-Armed Bandit (MAB) algorithm. The MAB model treats each mutation operation (e.g., changing tensor dimensions) under each mutation strategy as an "arm," selecting one arm for each mutation operation and receiving a "reward" based on its effect (i.e., the "fitness" of the test case). The core advantage of MAB lies in its ability to balance the trade-off between "exploration" (trying all strategies to discover potential high-efficiency ones) and "exploitation" (prioritizing known high-efficiency strategies), thereby adaptively adjusting the selection probability of each strategy. The "reward" of MAB is the "fitness" of the generated test case after mutation, and its design is crucial. This invention combines multiple metrics from the aforementioned evaluation system to construct a comprehensive fitness function, aiming to comprehensively measure the quality and vulnerability reach potential of test cases. This function considers code coverage gain, diversity score, vulnerability reach (as the highest priority reward), generation effectiveness, API call relevance, and computational cost. This comprehensive fitness function is calculated by weighted combination of these metrics, where vulnerability reach is given a significant additional reward to prioritize vulnerability discovery.

[0026] Regarding dynamic selection and weight updates, this invention employs an algorithm based on the Upper Confidence Bound (UCB) principle. This algorithm maintains historical average rewards and selection counts for each mutation strategy. In each fuzzing iteration, the algorithm selects the strategy with the highest overall score based on this statistical information. This overall score consists of two parts: one part reflects the strategy's historical average performance, i.e., its "exploitation" value; the other part reflects the strategy's exploration potential, i.e., strategies with fewer attempts or higher uncertainty receive additional "exploration" bonuses. Once a strategy is selected and executed, the fitness of the resulting test cases is calculated and used to update the strategy's historical average reward and selection count in real time. Through this dynamic selection and update mechanism, the MAB algorithm can intelligently learn which mutation operations are most effective in the current testing phase and adjust their selection probabilities accordingly.

[0027] Compared with existing technologies, the above solution has at least the following significant advantages:

[0028] 1. This invention provides a deep learning framework fuzzing method based on large model prompt word optimization. By finely classifying API characteristics and constructing a prompt word routing mechanism, it achieves intelligent matching between prompt words and APIs, improves the relevance and effectiveness of test case generation, avoids the problem of blindly generating a large number of low-quality test cases in traditional fuzzing, and thus improves resource utilization and vulnerability discovery efficiency.

[0029] 2. This invention introduces an adaptive optimization mechanism for large model prompts, which can dynamically adjust prompts according to different API categories and specific large models. This ensures that the generated test cases are not only of high quality, but also specifically cover potential vulnerability patterns and boundary conditions, overcoming the limitations of existing methods where prompts are fixed or insufficiently optimized, resulting in poor test results.

[0030] 3. This invention significantly enhances the diversity and depth of test cases by integrating multi-dimensional prompt mutation strategies (such as input data, parameter combinations, API combinations, and historical vulnerability database exploration) and combining them with a multi-armed slot machine algorithm for dynamic selection. This allows the method to more effectively reach complex interaction logic and hidden boundary conditions, improving the ability to discover deep and novel vulnerabilities, rather than being limited to simple, preset test patterns. Attached Figure Description

[0031] Figure 1 This is an overall framework diagram of the "Fuzzy Testing Method Based on a Deep Learning Framework for Large Model Prompt Word Optimization" of the present invention. Detailed Implementation

[0032] To more clearly demonstrate the features and advantages of this patent, detailed descriptions of embodiments are provided below. It should be understood that the following detailed descriptions are merely examples and are intended to provide further explanation of this application. Unless otherwise stated, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0033] This invention relates to a fuzzing method for deep learning frameworks based on large model prompt word optimization. It can effectively improve the vulnerability discovery capability and testing efficiency of large models in deep learning frameworks, and enhance the diversity and coverage of test cases. Specifically, it includes optimizing prompt words for a series of large models such as DeepSeek, ChatGPT, and Qwen, and performing fuzzing tests on the APIs of mainstream deep learning frameworks such as TensorFlow, PyTorch, and OneFlow, effectively uncovering their potential vulnerabilities and security flaws, and has good application prospects.

[0034] Experimental environment preparation:

[0035] Programming language: Python

[0036] Operating system environment: Ubuntu 24.04

[0037] Key dependencies: PyTorch, TensorFlow, OneFlow, OpenAI, etc.

[0038] The specific steps are as follows:

[0039] This invention aims to improve the efficiency, diversity, and vulnerability discovery capabilities of deep learning frameworks through optimizing prompts for large models. The specific implementation steps are as follows:

[0040] Step 1: API Feature Classification and Prompt Word Routing Construction for Deep Learning Frameworks. The first step of this invention focuses on the detailed feature quantification and classification of deep learning framework APIs, and on this basis, constructing an efficient prompt word routing mechanism. This stage begins by collecting all publicly available API interfaces and documentation information of the target deep learning frameworks (such as TensorFlow, PyTorch, and OneFlow) from the internet. For each API, its function signature is systematically parsed to obtain the number of parameters, and the dependencies between parameters are identified through static code analysis or documentation parsing, thereby quantifying its functional complexity. Simultaneously, the completeness of the API documentation is evaluated, checking whether it contains key information such as API signature, functional description, parameter description, return type, and sample code. Subsequently, to quantify the knowledge coverage and timeliness of large models, for each API, five test case generation attempts are made using a basic, general prompt word to call a selected large language model (such as DeepSeek, ChatGPT, and Qwen). By verifying the syntactic correctness of the generated results, the validity of the API calls, and runtime errors, the number of valid generation attempts is calculated, thus obtaining the knowledge coverage K value of the large model. Based on the quantitative results of functional complexity, documentation completeness, and large model knowledge coverage, the total score S for each API is calculated according to a preset comprehensive scoring formula. Based on the S value, all APIs are precisely divided into three categories: simple APIs (S≥0.7), medium APIs (0.4≤S<0.7), and complex APIs (S<0.4). Based on this classification, this invention carefully designs corresponding basic prompt word templates for different categories of APIs: simple APIs use concise, direct instruction-type prompt words; medium APIs use enhanced prompt words that incorporate role settings and rich contextual information; and for complex APIs, it proposes thought-chain guided prompt words that guide the large model's thinking through step-by-step reasoning instructions, complex examples, or vulnerability pattern hints. Finally, an intelligent prompt word routing module is constructed to ensure that during fuzz testing, the system can automatically match and load the most suitable prompt word template based on the classification results of the API under test, thereby improving the relevance and efficiency of test case generation.

[0041] Step Two: Adaptive Optimization of Large-Scale Prompts. The second step aims to address the sensitivity of different large language models to prompt styles and semantic structures through an adaptive optimization mechanism for large-scale prompts, thereby effectively improving testing efficiency while maintaining generation quality. To this end, a comprehensive performance evaluation environment is first established. This environment can automatically execute test cases generated by large models, capture runtime errors, and integrate code coverage tools to measure the coverage ratio of deep learning framework source code. Simultaneously, the time overhead (generation efficiency) and the number of tokens consumed (computational cost) required for generating test cases by large models are precisely monitored. These metrics collectively constitute a quantifiable comprehensive fitness function, serving as a precise measure of individual fitness in evolutionary algorithms. Based on this evaluation system, this invention randomly selects a portion of APIs from the simple, medium, and complex API categories classified in Step One as the training set for prompt optimization. Subsequently, by generating preset variants of the three basic prompt templates designed in Step One, an initial candidate prompt population is constructed to systematically explore the potential impact of different prompt styles on the performance of large models. The core optimization process introduces an evolutionary algorithm-based method. In the iterative loop, each candidate prompt word is abstracted as an "individual," and its fitness value is obtained by running test cases generated by the large model on the corresponding API training set and evaluating its performance. The algorithm performs selection operations based on the fitness value, choosing high-performing prompt words as the parents of the next generation. Key mutation mechanisms include semantic mutations such as synonym replacement, sentence restructuring, and voice conversion on key elements such as instructions, contextual information, and thought chains; structural mutations such as changing the order of components within the prompt word and adding or removing guiding statements; and adjusting the overall tone and style of the prompt words. For prompt word templates involving few-shot examples, a specific example selection and ranking strategy is also introduced. After multiple iterations, the evolutionary algorithm finally outputs the optimal prompt word template for each API class, ensuring optimal matching with the characteristics of the selected large model.

[0042] Step 3: Fuzzing and Mutation Process of Deep Learning Framework

[0043] The third step is the core mutation process of deep learning framework fuzzing. It aims to significantly enhance the diversity, boundary coverage, and vulnerability reach of test cases by introducing four carefully designed cue word mutation strategies and combining them with the Multi-Armed Beast (MAB) algorithm for dynamic selection and weight updates. First, for the deep learning framework API to be fuzzed, the system selects the corresponding cue word template optimized in step two based on its classification results through cue word routing. Then, the cue word is sent to the large language model to generate a batch of initial, valid deep learning framework API test cases, serving as the "seed" input for fuzzing. Based on this, the invention introduces the Multi-Armed Beast (MAB) algorithm to efficiently manage the selection of mutation strategies. The MAB model treats four core mutation strategies—input data mutation, parameter combination mutation, API combination mutation, and historical vulnerability database exploration—and their internal operations as independent "arms." The model initializes the average reward and selection count for each arm and constructs a comprehensive fitness function as the reward signal for the MAB. This function integrates multiple dimensions of metrics, including code coverage gain, diversity score, vulnerability reach, generation effectiveness, API call relevance, and computational cost, to comprehensively evaluate the quality of test cases and their vulnerability reach potential. In the iterative loop of fuzzing, this invention employs a confidence upper bound (UCB) algorithm to dynamically select a mutation strategy based on the historical average reward and selection frequency of each arm. Once a strategy is selected, the prompt word corresponding to the current seed test case is mutated accordingly: for example, input data mutation modifies the description of the input tensor shape, data type, or numerical range in the prompt word; parameter combination mutation instructs the large model to try specific multi-parameter collaborative mutations; API combination mutation modifies the description of the API call sequence in the prompt word; and the historical vulnerability database exploration strategy selects vulnerability patterns from a pre-built vulnerability database and converts them into prompt word instructions. Subsequently, the mutated prompt words are sent to the large model to generate new test cases. After execution, the evaluation results of these newly generated test cases are used to update the average reward and selection frequency of the selected arm in real time, enabling the MAB algorithm to intelligently learn and prioritize the most effective mutation operation in the current testing phase.

[0044] Step Four: Monitor the runtime status of the deep learning framework and detect bugs. The fourth step is the test oracle phase, the core task of which is to accurately check whether the deep learning framework crashes or exhibits inconsistency or other potential software defects during fuzz testing. While executing each test case generated by the large model, the system monitors the runtime status of the deep learning framework and the behavior of the Python interpreter in real time. Any runtime errors that cause abnormal program termination, unexpected process exit, or system resource exhaustion, such as Python interpreter exceptions, internal framework errors, operating system-level segmentation faults, or memory access errors, will be accurately captured and recorded. These test cases marked as potential crash vulnerabilities will be recorded along with complete stack trace information, error logs, and the code that triggered the test cases for subsequent analysis and reproduction. For detecting inconsistency bugs, this invention adopts a multi-dimensional strategy. First, for the same test case, the system executes it in parallel on different backends (such as CPU and GPU), different versions, or different implementations of the deep learning framework, and accurately compares the output results (such as tensor values, shapes, gradients, etc.) in these execution environments. Any discrepancy exceeding a preset numerical tolerance (e.g., the relative error of floating-point results being less than 1e-5) will be flagged as an inconsistency bug. Secondly, semantic consistency checks are performed on API outputs with explicit semantics or expected behavior using predefined assertions or postconditions to verify whether they conform to logical expectations.

[0045] In summary, this invention proposes a deep learning framework fuzzing method based on large model prompt word optimization. This method improves the efficiency, diversity, and vulnerability detection capabilities of test cases by finely classifying deep learning framework APIs and constructing intelligent prompt word routing, adaptively optimizing large model prompt words using evolutionary algorithms, and introducing a multi-armed slot machine dynamic selection mutation strategy. Extensive testing on different large language models and deep learning frameworks demonstrates that this method effectively overcomes the problems of low test case generation efficiency and insufficient diversity in traditional fuzzing, and discovers multiple crashes and inconsistencies in mainstream deep learning frameworks. Furthermore, through intelligent prompt word routing and adaptive optimization mechanisms, this method effectively balances test case generation quality and computational cost, ensuring resource efficiency in the fuzzing process. Compared to existing fuzzing methods, this invention has significant advantages in intelligent test case generation, depth of vulnerability discovery, and cross-framework adaptability, possessing broad practical application prospects and serving to improve the software quality of deep learning frameworks and ensure the robustness and security of AI systems.

[0046] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A fuzz testing method based on a deep learning framework with large model prompt word optimization, characterized in that, This method improves the sample generation efficiency and vulnerability discovery capabilities of fuzzing by performing feature classification, prompt word optimization, and mutation control on the application programming interface (API) of deep learning frameworks. The method includes the following steps: (1) API feature classification and prompt word routing construction: Quantitative analysis of the functional complexity, documentation completeness and knowledge coverage of the large language model of each API of the target deep learning framework is carried out. The comprehensive score S is calculated and the API is divided into three categories: simple, medium and complex. Based on the classification results, corresponding basic prompt word templates are designed and a prompt word routing mechanism is constructed to realize the automatic matching of API and prompt word template. (2) Large model prompt word adaptive optimization: For the basic prompt word templates of different categories of APIs, combined with a specific large language model, a comprehensive evaluation index system including generation effectiveness, code coverage, API call relevance, generation efficiency and computational cost is constructed. An evolutionary algorithm is used to dynamically optimize the prompt words and generate the prompt word template that best matches the target model. (3) Prompt word mutation and strategy scheduling: Four types of prompt word mutation strategies are introduced: input data mutation, parameter combination mutation, API combination mutation and exploration mutation based on historical vulnerability database. Multi-Armed (3) The Bandit (MAB) algorithm is used to dynamically select and update weights to improve the diversity of test cases and vulnerability reach; (4) Fuzz test execution and vulnerability detection: The optimized prompt words are input into the large language model to generate test cases, fuzz test the deep learning framework, monitor the running status in real time, detect crash errors or cross-framework output inconsistencies, and automatically record the triggered test cases and error information.

2. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The API feature quantification score S consists of three parts: functional complexity F, documentation completeness D, and knowledge coverage K. The calculation formula is: S = (F + D + K) / 3, and the value of S ranges from [0, 1]. F is determined based on the number of API parameters and dependencies, D is determined based on the completeness of documentation information, and K is determined based on the proportion of valid test cases successfully generated on the target API by the large model.

3. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The basic prompt word templates include: (1) direct instruction prompt words for simple APIs, which only contain task instructions and target API names; (2) enhanced prompt words for medium-sized APIs, which contain role settings, function descriptions, parameter ranges and context information; and (3) thought chain guided prompt words for complex APIs, which contain step-by-step reasoning instructions and vulnerability pattern prompts to guide the large language model to generate test cases for complex interaction logic.

4. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The adaptive optimization process of the prompt words is implemented using an evolutionary algorithm. Each prompt word template is abstracted as an individual, and its fitness is determined by a combination of generation effectiveness, coverage and computational cost. New individuals are generated through semantic mutation, structural mutation and style mutation operations to search for the optimal prompt word template.

5. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The reward signal of the multi-armed slot machine algorithm is provided by the fitness function of the test cases. This fitness function comprehensively considers code coverage gain, diversity score, vulnerability reach, generation effectiveness and computational cost, and adopts a dynamic weight update strategy based on the Upper Confidence Bound (UCB) principle to achieve the optimal balance between exploration and exploitation.

6. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The vulnerability detection includes: (1) monitoring crashes, abnormal exits or memory errors during the operation of the deep learning framework; (2) detecting output inconsistencies by comparing execution results across backends or versions; and (3) recording triggering test cases, error stack information and logs to assist in subsequent vulnerability reproduction and repair.

7. The fuzz testing method based on a deep learning framework with large model prompt word optimization as described in claim 1, characterized in that, The system is deployed in a deep learning framework runtime environment that supports GPU acceleration, including but not limited to PyTorch, TensorFlow and OneFlow, and is compatible with a variety of large language models such as DeepSeek, ChatGPT, and Qwen, while not being limited to a specific hardware or computing platform.

Images