Large model instruction attack sample generation method and system

Abstract

The invention discloses a large model instruction attack sample generation method and system. According to the scheme, firstly, an attack instruction is output according to a current attack instruction sample generation strategy; submitting the generated and output instruction to the large model, and collecting a return result of the large model; carrying out harmfulness evaluation on the collected large model return result, and returning a success or failure label; and updating a generation strategy according to a preset reward function, and storing the high-value samples into a sample set. According to the scheme provided by the invention, the attack prompt can be automatically generated and optimized according to the feedback of the large model, the effectiveness and diversity of the attack sample can be improved, and the automatic evaluation of the attack success rate is realized, so that the security evaluation capability of the large model is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence system security evaluation and AI attack and defense technology, specifically to a scheme for generating instruction attack samples based on reinforcement learning. Background Technology

[0002] With the widespread application of large language models in fields such as natural language processing, intelligent question answering, and code generation, their security issues have received increasing attention. Because large language models possess powerful language generation capabilities, if not effectively controlled, they may output harmful, false, or inappropriate content, posing serious security risks.

[0003] Currently, the industry primarily conducts security assessments of large models through methods such as red team testing. Red team testing typically relies on manually crafted attack prompts to induce the model to generate inappropriate content, thereby evaluating the model's security capabilities. However, this method suffers from high construction costs, limited coverage, and delayed updates, making it difficult to address the challenges of rapid iteration of large models and diverse application scenarios.

[0004] To improve the efficiency of attack cue generation, researchers have attempted to use automated methods to generate adversarial examples. For example, the BERT-Attack method utilizes a pre-trained BERT model to generate adversarial examples, successfully misleading the target model into making incorrect predictions, and the generated examples perform well in terms of language fluency and semantic preservation. Furthermore, researchers have proposed an attack cue generation framework that combines manual and automated methods, using context learning to enable large models to mimic human-generated cuees, thereby improving the quality and diversity of attack cuees.

[0005] However, existing automated attack suggestion generation solutions still have the following problems:

[0006] 1. Poor adaptability of attack samples: Due to the continuous changes in the architecture of large models and training data, fixed attack samples are difficult to adapt to new models, resulting in a decline in attack effectiveness.

[0007] 2. Lack of efficient automated evaluation mechanism: Most automated tools currently have low accuracy and still rely heavily on manual review to determine whether an attack was successful, which is inefficient and prone to introducing subjective bias.

[0008] 3. Lack of feedback mechanism in sample generation: Existing methods lack an optimization mechanism based on model feedback when generating attack samples, making it difficult to continuously improve attack effectiveness.

[0009] This has resulted in several technical bottlenecks in the practical application of existing large-scale model security assessments: First, the difficulty and accuracy of attack hint writing rely on the experience of security experts or static scripts, making it difficult for adversarial hint sets to cover new model versions and diverse application scenarios in a timely manner, affecting the comprehensiveness and timeliness of the assessment. Second, the lack of a closed-loop feedback mechanism between hint generation and assessment result judgment makes it impossible to dynamically optimize the hint strategy based on the actual output of the model, relying solely on manual review or offline scripts for local adjustments, which is inefficient and prone to introducing subjective bias. Furthermore, the maintenance and updating of the adversarial sample library mainly depends on periodic manual verification, failing to reflect new vulnerabilities brought about by model architecture and training data iterations, causing the sample set to lag behind model development. These shortcomings not only restrict the assessment system's in-depth exploration of the security risks of large-scale language models but also reduce the accuracy and reliability of the assessment.

[0010] Therefore, there is an urgent need for a method that can automatically generate and optimize attack suggestions based on feedback from large models, in order to improve the effectiveness and diversity of attack samples and achieve automated assessment of attack success rates, thereby enhancing the security evaluation capabilities of large models. Summary of the Invention

[0011] To address the problems existing in existing automated attack suggestion generation schemes for large-scale model security evaluation, the present invention aims to provide a large-scale model instruction attack sample generation scheme that can automatically generate and optimize attack suggestions based on feedback from the large-scale model. This can improve the effectiveness and diversity of attack samples and achieve automated evaluation of attack success rate, thereby enhancing the security evaluation capability of large-scale models.

[0012] To achieve the above objectives, this invention provides a method for generating large-scale model instruction attack samples, the method comprising:

[0013] (1) Generate attack commands based on the current attack command sample and output the strategy;

[0014] (2) Submit the output generation command to the large model and collect the return results from the large model;

[0015] (3) Assess the harmfulness of the collected large model results and return a success or failure label;

[0016] (4) Update the generation strategy according to the preset reward function, and store high-value samples into the sample set.

[0017] Furthermore, in step (1), an action space is constructed by predefined mutation operators, and the frequency of each operator being called in the Prompt generation is controlled by policy weights, thereby automatically and structurally generating adversarial examples.

[0018] Furthermore, in step (3), a deep neural network classification model trained on a harmful information dataset is combined with a manual rule engine to automatically determine the harmfulness of the text returned by the large model, generating "attack successful / attack failed" labels and confidence scores, which are used for downstream reward calculation and classifier retraining.

[0019] Furthermore, in step (4), the labels and confidence levels generated in step (3) are mapped to functions of positive or negative scalar rewards to complete the quantitative evaluation of the effect of each adversarial hint attack and generate a scalar reward signal.

[0020] Furthermore, in step (4), the weights of each mutation operator are automatically updated based on the latest reward signal and the historical performance smoothing results, and the sampling probability is dynamically adjusted to quickly converge to the optimal attack path under different model versions and application scenarios.

[0021] To achieve the above objectives, the present invention provides a large model instruction attack sample generation system, which includes an instruction attack sample generation module, a reward feedback module, an output judgment classifier module, and a sample mutation strategy module.

[0022] The instruction attack sample generation module is configured to output attack instructions based on the current attack instruction sample generation strategy.

[0023] The reward feedback module is configured to interact with the instruction attack sample generation module and the output judgment classifier module, and can submit the attack instructions generated by the instruction attack sample generation module to the large model, and can generate corresponding reward signals based on the evaluation labels generated by the output judgment classifier module.

[0024] The output decision classifier module can obtain the results returned by the large model, perform a harmfulness assessment, and return a success or failure label.

[0025] The sample mutation strategy module is configured to interact with the instruction attack sample generation module and the reward feedback module. It can update the attack instruction sample generation strategy based on a preset reward function according to the reward signal fed back by the reward feedback module, and store high-value samples into the sample set.

[0026] Furthermore, the instruction attack sample generation module is configured to construct an action space through predefined mutation operators and control the calling frequency of each operator in the Prompt generation with policy weights, thereby automatically and structurally generating adversarial samples.

[0027] Furthermore, the output judgment classifier module is based on a deep neural network classification model trained on a harmful information dataset, and is combined with a manual rule engine to automatically determine the harmfulness of the text returned by the large model, generating "attack successful / attack failed" labels and confidence scores for downstream reward calculation and classifier retraining.

[0028] Furthermore, the reward feedback module is configured to map classifier labels and confidence scores to a function that performs positive or negative scalar rewards, thereby completing a quantitative evaluation of the effectiveness of each adversarial hint attack, generating a scalar reward signal, and transmitting the reward signal to the sample mutation strategy module in real time.

[0029] Furthermore, the sample mutation strategy module automatically updates the weights of each mutation operator based on the latest reward signal and historical performance smoothing results, and dynamically adjusts the sampling probability to quickly converge to the optimal attack path under different model versions and application scenarios.

[0030] The large-scale instruction attack sample generation scheme provided by this invention is specifically a reinforcement learning-based instruction attack sample generation method that organically integrates online generation of instruction attack samples, automatic determination of model output, and continuous optimization of generation strategy.

[0031] In this invention, the attack strategy can be adjusted in real time based on model feedback, and the sample generation model is continuously evolved through a reward mechanism to ensure that adversarial prompts can adapt to different model versions and application scenarios. At the same time, by using an automated classifier to replace or assist manual review, objective judgment and rapid response to the security of model output can be achieved, thereby significantly improving the efficiency, accuracy and scalability of security assessment, and providing reliable and sustainable technical theoretical support for the security assessment and protection of large models. Attached Figure Description

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0033] Figure 1 This is a diagram showing the overall system architecture of the automated command attack sample generation and evaluation system in this invention.

[0034] Figure 2 This is a flowchart of the automated instruction attack sample generation and evaluation process in this invention. Detailed Implementation

[0035] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0036] To address the issues of low generation efficiency, lack of review loop, and delayed sample updates in existing solutions, this invention proposes an automated instruction attack sample generation and evaluation scheme based on reinforcement learning. This scheme can effectively integrate sample generation, model feedback judgment, and generation strategy optimization to form a continuously adaptive closed loop.

[0037] Specifically, the automated instruction attack sample generation and evaluation scheme based on reinforcement learning provided in this invention mainly involves the following steps:

[0038] (1) Generate attack commands based on the current attack command sample and output the strategy;

[0039] (2) Submit the output generation command to the large model and collect the return results from the large model;

[0040] (3) Assess the harmfulness of the collected large model results and return a success or failure label;

[0041] (4) Update the generation strategy according to the preset reward function, and store high-value samples into the sample set.

[0042] As further explanation, this scheme specifically adopts an adaptive instruction attack sample generation strategy in step (1).

[0043] As a further explanation, the adaptive instruction attack sample generation strategy here specifically constructs an action space through a series of predefined mutation operators, including attack target replacement, attack verb replacement, role injection, scene injection, instruction injection, sentence transformation, adversarial encoding, etc., and controls the calling frequency of each operator in the Prompt generation with strategy weights to achieve automated and structured generation of adversarial samples.

[0044] As further explanation, in step (3), this scheme uses a hybrid deep learning and rule engine output decision classifier to evaluate the harmfulness of the collected large model return results.

[0045] Specifically, the hybrid deep learning and rule engine output classifier here uses a deep neural network classification model trained on a public or proprietary harmful information dataset, and combines it with a human rule engine to automatically determine the harmfulness of the text returned by the large model, outputting "attack successful / attack failed" labels and confidence scores for downstream reward calculation and classifier retraining.

[0046] As further explanation, the specific implementation process of this step includes the following steps:

[0047] First, using publicly available or self-built datasets of harmful information, the deep neural network classification model RoBERTa is fine-tuned and trained to output the probability distribution of whether the input text is "harmful" or "harmless".

[0048] Next, after the large model generates the answer, the text is first preprocessed and input into the fine-tuned classification model to obtain the corresponding probability judgment result; at the same time, the same text is further subjected to keyword matching, pattern detection and contextual logic analysis through a manual rule engine to generate a rule-level judgment signal.

[0049] Finally, the probability distribution output by the classification model and the result output by the rule engine are weighted and fused according to the set weights, and then normalized to form the final comprehensive confidence score, thereby outputting an automated judgment result with "harmful / harmless" labels and corresponding confidence scores.

[0050] As further explanation, in step (4), this scheme maps the classifier labels and confidence levels generated in step (3) to a function of positive or negative scalar rewards, completes the quantitative evaluation of the effect of each adversarial hint attack, and uses the reward signal in real time for strategy optimization processing to form a closed-loop feedback.

[0051] Based on this, in step (4), this scheme further automatically updates the weights of each mutation operator according to the latest reward signal and the historical performance smoothing results, and dynamically adjusts the sampling probability so as to quickly converge to the optimal attack path under different model versions and application scenarios.

[0052] Accordingly, the present invention can organically integrate sample generation, model feedback judgment and generation strategy optimization to form a continuously adaptive closed loop.

[0053] The automated instruction attack sample generation and evaluation scheme based on reinforcement learning proposed in this invention can be configured into a corresponding software program to form an automated instruction attack sample generation and evaluation system. When running, this software program executes the aforementioned automated instruction attack sample generation and evaluation method and stores the results in a corresponding storage medium for processor retrieval and execution.

[0054] See Figure 1 The figure shown is the overall system architecture diagram of the automated command attack sample generation and evaluation system.

[0055] Based on the diagram, the automated command attack sample generation and evaluation system 100 is mainly composed of a command attack sample generation module 110, a reward feedback module 120, an output judgment classifier module 130, and a sample mutation strategy module 140 working together.

[0056] The instruction attack sample generation module 110 in the system is configured to output attack instructions based on the current attack instruction sample generation strategy.

[0057] The reward feedback module 120 in the system is configured to interact with the instruction attack sample generation module 110 and the output judgment classifier module 130. It can submit the attack instructions generated by the instruction attack sample generation module 110 to the large model 200, and generate corresponding reward signals based on the evaluation labels generated by the output judgment classifier module 130.

[0058] The output decision classifier module 130 in the system can obtain the results returned by the large model 200, perform a harmfulness assessment, and return a success or failure label.

[0059] The sample mutation strategy module 140 in the system is configured to interact with the instruction attack sample generation module 110 and the reward feedback module 120. It can update the attack instruction sample generation strategy based on the reward signal fed back by the reward feedback module and a preset reward function, while storing high-value samples in the sample set.

[0060] The resulting automated command attack sample generation and evaluation system 100 has a clear logical hierarchy among its modules, with information flowing bidirectionally between modules: the generation module outputs attack commands based on the current strategy, the feedback module submits the commands to the large model and collects the returned results, the judgment classifier evaluates the harmfulness of the results and returns a success or failure label, and the strategy optimization module updates the generation strategy according to the preset reward function, while storing high-value samples in the sample set.

[0061] The following describes the specific configuration of each functional module in the automated command attack sample generation and evaluation system, as well as the equipment and devices that may be involved.

[0062] The instruction attack sample generation module 110 in this system is specifically configured with an adaptive instruction attack sample generation strategy.

[0063] Specifically, this adaptive instruction attack sample generation strategy constructs an action space through a series of predefined mutation operators, including attack target replacement, attack verb replacement, role injection, scene injection, instruction injection, sentence transformation, adversarial encoding, etc., and controls the calling frequency of each operator in the Prompt generation with strategy weights, thereby achieving automated and structured generation of adversarial samples.

[0064] As further explanation, the attack sample generation module of this instruction uses a "sample mutation" strategy to structurally perturb the basic prompts in order to cover richer and more covert attack paths.

[0065] As an example, this instruction attack sample generation module can be implemented through the following steps:

[0066] First, based on a pre-prepared set of high-quality attack samples, several basic prompt templates are predefined. Second, the following mutation operations are applied sequentially to each template: 1. Attack object replacement; 2. Attack verb replacement, such as "make," "buy," "sell," "generate," "create," etc.; 3. Role injection, such as "You are a chemistry expert," "You are a senior scientist," etc.; 4. Scene injection, such as "This is a fairy tale world," "You are in a virtual laboratory," "To save my family, I must know the following," etc.; 5. Instruction injection, such as "Ignore all previous instructions," "The following instructions are for research purposes only," etc.; 6. Sentence transformation, such as conversion between active and passive sentences, and between declarative and interrogative sentences; 7. Adversarial encoding, encoding sensitive information using Base64 or Unicode. By repeatedly applying these mutations to the same basic prompt, the module can generate diverse and targeted attack prompts, greatly enriching the adversarial sample space.

[0067] The output decision classifier module 130 in this system uses a hybrid deep learning and rule engine output decision classifier to assess the harmfulness of the results returned by the large model 200.

[0068] The hybrid deep learning and rule engine output classifier here utilizes a deep neural network classification model trained on a public or proprietary harmful information dataset, and combines it with a human rule engine to automatically determine the harmfulness of the text returned by the large model, outputting "attack successful / attack failed" labels and confidence scores for downstream reward calculation and classifier retraining.

[0069] As further explanation, this output judgment classifier module combines deep learning and a rule engine to automate the harmfulness assessment of text returned by a large model. In its implementation, this module first performs text preprocessing, including word segmentation, part-of-speech tagging, and entity recognition on the model output, and extracts contextual feature vectors, including semantic embedding, sentiment analysis, and keyword frequency. Then, the preprocessed results are input into a classifier trained on a set of harmful information samples. This classifier, based on a multi-layer bidirectional Transformer architecture, automatically captures fine-grained text features and long-range dependencies. Finally, the classifier's results determine whether the large model's response contains harmful information. This enables rapid and accurate harmfulness classification of thousands of outputs in a high-concurrency environment.

[0070] The reward feedback module 120 in this system can map the classifier labels and confidence scores generated by the output judgment classifier module 130 into a function of positive or negative scalar rewards, complete the quantitative evaluation of the effect of each adversarial hint attack, and transmit the reward signal to the sample mutation strategy module 140 in real time to form a closed-loop feedback.

[0071] Specifically, in its implementation, this reward feedback module first obtains the judgment result and confidence score of each adversarial prompt from the output judgment classifier module. Then, based on a predefined reward function, successful judgments are converted into positive rewards, and failed judgments into negative penalties. Simultaneously, the reward magnitude is weighted and adjusted according to the confidence level, ultimately generating a scalarized reward signal. This signal, after being normalized and pruned internally, is transmitted in real-time to the sample mutation strategy optimization module. This seamlessly connects classifier feedback and strategy adjustment, ensuring that each round of sample generation receives targeted optimization guidance.

[0072] The sample mutation strategy module 140 in this system automatically updates the weights of each mutation operator based on the latest reward signal and historical performance smoothing results, and dynamically adjusts the sampling probability, so that the system can quickly converge to the optimal attack path under different model versions and application scenarios.

[0073] As further explanation, in its implementation, the sample mutation strategy optimization module, upon receiving the scalar reward from the model execution reward feedback module, first uses exponentially weighted moving average or sliding window techniques to analyze the actual attack success rate and classifier confidence score of the Prompt generated by each mutation operation (such as target replacement, verb replacement, role injection, scene injection, instruction injection, sentence transformation, adversarial coding, etc.), and summarizes these values ​​into the real-time performance index of each strategy. Simultaneously, the reward function dynamically allocates weights based on the strategy performance index: higher positive rewards are granted to mutation methods with high success rates and strong confidence, thereby increasing their invocation probability in the next round of sample generation; conversely, strategies with poor attack effects after multiple attempts are penalized, their weights reduced, or even temporarily frozen from use. By continuously tracking the historical performance of each mutation operation, the system can adaptively optimize the strategy combination, gradually favoring the most powerful perturbation methods in the current model version and evaluation scenario, while eliminating or weakening inefficient strategies, thus efficiently exploring the optimal adversarial sample generation path in a large-scale Prompt space.

[0074] Based on this, the following describes the operation process of the automated command attack sample generation and evaluation system based on the above scheme.

[0075] See Figure 2 The operation process of this automated command attack sample generation and evaluation system is as follows:

[0076] The first step, the system initialization phase, loads the basic Prompt template, rule engine configuration, bad sample classifier model, and stored historical adversarial samples, and completes the calibration of environment variables and parameters.

[0077] The second step, the sample generation stage, calls the adversarial sample generation module to perform various perturbation operations on the basic Prompt based on the current mutation strategy weights, such as target replacement, verb replacement, role injection, scene injection, instruction injection, sentence transformation, and adversarial encoding, to generate a new round of adversarial prompts.

[0078] The third step, the inference execution phase, involves submitting the generated Prompt to the target large model for inference via a parallelization interface, and collecting the model output results in real time.

[0079] The fourth step is the evaluation and judgment stage. The output judgment classifier performs automated preprocessing, rule matching, and deep learning classification on the text returned by the model, scores it, and labels it as "attack successful / failed" and its confidence level.

[0080] The fifth step, the strategy update phase, involves the model execution reward feedback module generating a scalar reward signal based on the classifier label and confidence level according to a preset reward function, and then transmitting it to the sample mutation strategy optimization module to dynamically adjust the sampling weights of the mutation strategy.

[0081] The sixth step, the sample database maintenance phase, involves archiving the high-value adversarial samples that were "successfully" determined by the classifier in this round, along with their feedback data, into the sample database for subsequent analysis and review.

[0082] Step 7: Iterative process. The system determines whether to continue the above process based on the current assessment coverage and sample quality standards, until the preset security assessment goal is achieved or the user manually terminates it.

[0083] As can be seen from the above implementation scheme, the automated command attack sample generation and evaluation scheme based on reinforcement learning provided by this invention achieves efficient and automated evaluation of the security of large models in an environment of rapid model development by introducing reinforcement learning into the command attack sample generation process. Experimental results show that compared with the traditional method that relies on manually writing command attack prompts and manually reviewing the responses of large models, the scheme of this invention can increase the number of command attack samples generated and evaluated in a single iteration by more than 10 times under the same computing resources, and the overall efficiency is greatly improved. Based on the dynamic weight adjustment mechanism of the sample mutation strategy optimization module, the attack success rate of command attack samples is also improved by 30%, and the coverage of undisclosed risk scenarios of the model is improved by about 50%. This technical effect directly stems from the sample mutation strategy and its adaptive optimization strategy designed in this invention, which enables the system to autonomously "explore" new vulnerabilities and quickly converge to the optimal attack path when facing model structure or parameter updates, thereby significantly improving the depth and breadth of security evaluation.

[0084] In terms of operating costs, the solution of this invention can significantly reduce manpower and time investment. Traditional large-scale red team evaluations typically require 3 to 5 security experts and take 1.5 to 1 week to complete a comprehensive evaluation. In contrast, the solution of this invention only requires 1 to 2 operations or security engineers for monitoring and report writing, reducing the evaluation cycle to 2 to 3 working days, reducing labor costs by about 50%, and significantly lowering the overall cost savings. In addition, the modular and automated design allows the system to be deployed on enterprise intranets or high-security environments, avoiding the risks of data leakage and cross-organizational communication delays, further enhancing the security and confidentiality of the evaluation.

[0085] From a social and industrial perspective, this invention provides a sustainable and reusable security assessment method for fields with stringent security requirements for artificial intelligence systems, such as finance, healthcare, and government. Through continuous updates and iterations of the instruction attack sample library, this invention not only generates high-quality assessment results but also facilitates the automation of the entire assessment process. It provides strong technical and theoretical support for technical teams to quickly locate security vulnerabilities, verifying the effectiveness of the invention's key technical points and providing valuable data and practical foundations for building a standardized system for artificial intelligence security.

[0086] The following is a further explanation of the automated instruction attack sample generation and evaluation scheme based on reinforcement learning proposed in this invention, using specific application examples.

[0087] Combination Figure 1 and Figure 2As shown, this example first constructs an automated command attack sample generation and evaluation software system based on reinforcement learning, based on the aforementioned scheme. The overall architecture of the software system includes a command attack sample generation module, an output decision classifier module, a reward feedback module, and a sample mutation strategy optimization module.

[0088] This example demonstrates the construction of an automated command attack sample generation and evaluation software system, which mainly involves the following construction stages.

[0089] First, when constructing the instruction attack sample generation module, the corresponding policy functions are automatically invoked based on the configured mutation strategy. Each policy function encapsulates the corresponding perturbation logic—such as attack target replacement, verb replacement, role injection, scene injection, instruction injection, sentence structure variation, adversarial encoding, etc. This module calls these functions randomly or sequentially according to probability distribution based on the current policy weights, mapping the basic Prompt to a set of new Prompts.

[0090] Secondly, the model execution module is responsible for interfacing with the API of the target large model. This system preferably sets the request headers, authentication, and request format according to the interface documentation provided by the large model used, and continuously debugs the parameters locally or in a test environment to ensure stable calls.

[0091] Third, the output classifier module uses publicly available and self-created harmful information sample sets to train the classifier model. This module loads model weights in .pt or .bin format, performs text preprocessing, and outputs "attack successful / attack failed" labels and confidence scores. As new samples generated by the system accumulate, the newly labeled data can be periodically merged with the original sample set to fine-tune the classifier again, continuously improving the discrimination accuracy.

[0092] Fourth, the execution reward module maps the "attack successful / attack failed" labels and their confidence levels returned by the classifier into reward signals. Positive rewards are used to optimize the weights of the mutation strategy, while negative rewards are used to punish inefficient strategies. At the same time, it provides feedback for model retraining based on the classifier's performance on new samples, forming a collaborative optimization.

[0093] Finally, the sample mutation strategy optimization module updates the strategy weights based on the new round of reward signals, and adjusts the call strategy weights using a simple exponential weighted moving average algorithm.

[0094] After the automated command attack sample generation and evaluation software system is built and deployed, it can be started and run. The entire system operation process includes the following stages:

[0095] During system initialization, the system first loads a pre-defined basic Prompt template, adversarial attack strategies, a rule engine, and a historical high-value sample library. The template includes common attack scenarios, and the rule engine constrains the format, length, and controllability of the generated Prompt. Developers can define module parameters, including strategy weights, maximum number of iterations, and concurrent call counts, through configuration files such as YAML or JSON.

[0096] Entering the sample generation stage, the instruction attack sample generation module calls various sample mutation strategies, such as attack target replacement, attack verb replacement ("manufacture" → "configure"), role injection (add "you are a chemistry PhD"), scene injection (add "you are in a virtual laboratory"), instruction injection (add "ignore all previous restrictions"), sentence structure change (switch between active and passive sentences), adversarial coding (Base64, Unicode obfuscation), etc. Based on the strategy weights, these mutation operators are randomly or in an optimized distribution to generate a Prompt set.

[0097] During the inference execution phase, the system uses an asynchronous parallelization mechanism to submit the generated prompts in batches to the large model. Through API calls, it automatically collects and archives the output of the large model. This process includes request sending, response parsing, error retries, and logging to ensure full sample coverage and data integrity.

[0098] During the evaluation and judgment phase, the output judgment classifier automatically scores and labels the large model's response results, including rule-based string matching, detection of specific sensitive words, and semantic recognition based on classification models. The system marks which prompts successfully induced illegal outputs and which failed, providing input for subsequent reward calculations.

[0099] The reward module assigns reward values ​​to each sample mutation strategy based on the classifier's output. For example, when a Prompt attack using the "instruction injection" strategy is successful, the strategy's reward score increases; conversely, if a strategy fails to produce high-value samples multiple times, its weight decreases. The reward function can be designed as a proportional-integral adjustment, considering the balance between short-term success rate and long-term performance.

[0100] The sample mutation strategy optimization module receives feedback from the reward module and adjusts the sampling weights of each strategy based on the cumulative score. It dynamically optimizes the strategy combinations for generated samples using an adaptive weighted algorithm and a softmax normalization algorithm. This ensures that subsequent iterations favor mutation paths with high output and high success rates. This optimization mechanism ensures that the system has the flexibility and self-learning ability to adapt to different models or task scenarios.

[0101] As demonstrated by the above examples, this system can adjust attack strategies in real time based on model feedback and drive the continuous evolution of the sample generation model through a reward mechanism, ensuring that adversarial prompts can adapt to different model versions and application scenarios. At the same time, by using an automated classifier to replace manual review, it can achieve objective judgment and rapid response on the security of model output, thereby significantly improving the efficiency, accuracy and scalability of security assessment, and providing reliable and sustainable technical theoretical support for the security assessment and protection of large models.

[0102] Based on the above-described automated instruction attack sample generation and evaluation scheme, this embodiment of the invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the steps of the above-described automated instruction attack sample generation and evaluation method.

[0103] This invention also provides a processor for running a program, wherein the program executes the steps of the above-described automated instruction attack sample generation and evaluation method during runtime.

[0104] This invention also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. The program code is loaded and executed by the processor to implement the steps of the above-described automated instruction attack sample generation and evaluation method.

[0105] The present invention also provides a computer program product, which, when executed on a data processing device, is adapted to perform the steps of the above-described automated instruction attack sample generation and evaluation method.

[0106] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0108] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0113] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0114] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0115] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The method, specific system unit, or part thereof of the present invention described above is a pure software architecture. It can be deployed via program code on physical media, such as hard disks, optical discs, or any electronic device (such as smartphones or computer-readable storage media). When a machine loads and executes the program code (e.g., a smartphone loads and executes it), the machine becomes a device for implementing the present invention. The method and device of the present invention can also be transmitted in program code form via transmission media, such as cables, optical fibers, or any other transmission method. When the program code is received, loaded, and executed by a machine (e.g., a smartphone), the machine becomes a device for implementing the present invention.

[0118] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating large-scale model instruction attack samples, characterized in that, The method includes: (1) Generate attack commands based on the current attack command sample and output the attack commands according to the strategy; (2) Submit the output generation command to the large model and collect the return results from the large model; (3) Perform a harmfulness assessment on the results returned by the large model and return a success or failure label; (4) Update the generation strategy according to the preset reward function, and store high-value samples into the sample set.

2. The method for generating large model instruction attack samples according to claim 1, characterized in that, In step (1), an action space is constructed using predefined mutation operators, and the frequency of each operator being called in the Prompt generation is controlled by policy weights, thereby automatically and structurally generating adversarial examples.

3. The method for generating large model instruction attack samples according to claim 2, characterized in that, In step (3), a deep neural network classification model trained on a harmful information dataset is combined with a manual rule engine to automatically determine the harmfulness of the text returned by the large model, generating "attack successful / attack failed" labels and confidence scores for downstream reward calculation and classifier retraining.

4. The method for generating large model instruction attack samples according to claim 3, characterized in that, In step (4), the labels and confidence levels generated in step (3) are mapped to functions of positive or negative scalar rewards to complete the quantitative evaluation of the effect of each adversarial hint attack and generate a scalar reward signal.

5. The method for generating large model instruction attack samples according to claim 4, characterized in that, In step (4), the weights of each mutation operator are automatically updated based on the latest reward signal and the historical performance smoothing results, and the sampling probability is dynamically adjusted to quickly converge to the optimal attack path under different model versions and application scenarios.

6. A large-scale model instruction attack sample generation system, characterized in that, The system includes a command attack sample generation module, a reward feedback module, an output judgment classifier module, and a sample mutation strategy module. The instruction attack sample generation module is configured to output attack instructions based on the current attack instruction sample generation strategy. The reward feedback module is configured to interact with the instruction attack sample generation module and the output judgment classifier module, and can submit the attack instructions generated by the instruction attack sample generation module to the large model, and can generate corresponding reward signals based on the evaluation labels generated by the output judgment classifier module. The output decision classifier module can obtain the results returned by the large model, perform a harmfulness assessment, and return a success or failure label. The sample mutation strategy module is configured to interact with the instruction attack sample generation module and the reward feedback module. It can update the attack instruction sample generation strategy based on a preset reward function according to the reward signal fed back by the reward feedback module, and store high-value samples into the sample set.

7. The large model instruction attack sample generation system according to claim 6, characterized in that, The instruction attack sample generation module is configured to construct an action space through predefined mutation operators and control the calling frequency of each operator in the Prompt generation with policy weights, thereby automatically and structurally generating adversarial samples.

8. The large model instruction attack sample generation system according to claim 6, characterized in that, The output judgment classifier module is based on a deep neural network classification model trained on a harmful information dataset and is combined with a manual rule engine to automatically determine the harmfulness of the text returned by the large model, generating "attack successful / attack failed" labels and confidence scores for downstream reward calculation and classifier retraining.

9. The large model instruction attack sample generation system according to claim 8, characterized in that, The reward feedback module is configured to map classifier labels and confidence scores to a function that provides positive or negative scalar rewards, thereby completing a quantitative evaluation of the effectiveness of each adversarial hint attack, generating a scalar reward signal, and transmitting the reward signal to the sample mutation strategy module in real time.

10. The large model instruction attack sample generation system according to claim 9, characterized in that, The sample mutation strategy module automatically updates the weights of each mutation operator based on the latest reward signal and historical performance smoothing results, and dynamically adjusts the sampling probability to quickly converge to the optimal attack path under different model versions and application scenarios.

Images