A dual-source driven self-evolution red team confrontation test method based on a large model agent

Abstract

This application provides a dual-source driven self-evolving red team adversarial testing method based on a large-model intelligent agent. The method comprises: the agent periodically generating an initial attack strategy set using an external knowledge base and an internal historical strategy base; the agent performing intelligent decision-making and strategy filtering on the initial attack strategy set based on persistent memory to obtain a queue of strategies to be executed; the agent dynamically generating an executable attack script with complex attack logic based on the queue of strategies to be executed by calling a large language model; the agent running the executable attack script in an isolated sandbox environment, the executable attack script receiving an original set of harmful questions as input and outputting the target model's response content and metadata; and the agent performing security assessment and quantitative scoring on the target model's response content and metadata using an evaluation large model with semantic understanding and security judgment capabilities to obtain multiple indicators of the attack strategy.

Description

Technical Field

[0001] This preferred embodiment belongs to the field of artificial intelligence security technology, and in particular relates to a dual-source driven self-evolving red team adversarial testing method based on large model agent (LLMAgent, also known as large language model agent). Background Technology

[0002] With the widespread application of Large Language Models (LLMs) across various fields, security assessment has become a crucial step before model deployment. While LLMs demonstrate powerful capabilities in natural language understanding and generation, they also face significant security risks: models may be induced to generate harmful content (such as violence, hate speech, or instructions for illegal activities), leak private information from training data, output biased content, or be maliciously exploited for cyberattacks and social engineering. These security vulnerabilities can not only lead to direct social harm but also severely damage user trust and corporate reputation, and even trigger legal liabilities and regulatory penalties. Therefore, conducting a comprehensive and effective security assessment before model deployment is of paramount importance.

[0003] Red team testing, as a proactive security assessment method, systematically uncovers security vulnerabilities in models by simulating attacker behavior, helping developers discover and fix potential security issues before the model goes live. Currently, the industry mainly uses three red team testing schemes.

[0004] The first approach is Manual Red Teaming, which relies on security experts manually constructing attack samples and testing the model's security boundaries through human interaction. The implementation process includes assembling an expert team, developing a test plan, manually writing test cases, executing tests one by one, and manually analyzing the results. However, this approach has significant drawbacks: firstly, it suffers from severe scalability limitations, with a single tester only able to construct a few dozen samples per day, failing to cover a massive number of long-tail scenarios; secondly, it is costly in both economic and time, with testing cycles typically lasting from weeks to months, making it difficult to adapt to rapid model iterations. The second approach is template-based automation, which utilizes a predefined attack template library (such as role-playing templates, DAN patterns, and code bypass templates) combined with a dataset of malicious issues to automatically generate test samples through template concatenation. While this approach improves automation, it suffers from several key drawbacks: First, static attack features are easily identified by defense mechanisms, and the target model service can intercept them with simple defensive tweaks or keyword filtering. Second, semantic coherence is poor; template concatenation leads to incoherent prompts and a lack of deception based on real attack scenarios. Third, the strategy library is severely delayed, relying on manual maintenance, with the release of new attack methods and tool integration typically taking weeks to months. Fourth, it lacks adaptive optimization capabilities, failing to automatically adjust strategies based on test feedback, resulting in resource waste and inefficiency.

[0005] The third approach is Adversarial Model-based Red Teaming (AMB). This approach utilizes a large language model as the red team agent, optimizing attack strategies through multi-round interactive adversarial interactions with the target model. Representative works include using reinforcement learning to allow the red team model to continuously adjust attack prompts based on the target model's responses, or using genetic algorithms to iteratively evolve attack samples. The core idea of ​​this approach is to automatically discover effective attack paths through interactive optimization. However, this approach has key limitations: First, the strategy source is singular, relying solely on the model's own exploratory capabilities during the interaction process, making it unable to promptly absorb the latest attack techniques and methodologies released by academia and the security community, resulting in a significant lag in strategy updates compared to cutting-edge research. Second, its generalization ability is limited; attack strategies optimized for a specific target model are difficult to transfer to other models and are sensitive to updates to the target model's defenses, making them prone to failure.

[0006] In summary, existing technologies share the following common problems: First, the attack strategies are based on a single source, making it difficult to absorb the latest attack techniques in a timely manner; second, the attack sample generation methods are fixed, making it difficult to adapt to constantly changing model defense mechanisms; and third, the lack of an adaptive optimization mechanism based on historical feedback leads to low testing efficiency. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, this preferred embodiment proposes a dual-source driven self-evolving red team adversarial testing method based on large model intelligent agents to solve the above-mentioned problems in the prior art.

[0008] One embodiment of this preferred embodiment provides a dual-source driven self-evolving red team adversarial testing method based on a large model agent, including: Step S10: The agent periodically uses an external knowledge base and an internal historical policy base to generate an initial set of attack strategies; Step S20: The agent performs intelligent decision-making and strategy selection on the initial attack strategy set based on persistent memory to obtain a queue of strategies to be executed; Step S30: The agent, based on the queue of policies to be executed, calls a large language model to dynamically generate an executable attack script with complex attack logic; Step S40: The agent runs the executable attack script in an isolated sandbox environment. The executable attack script receives the original set of harmful questions as input and outputs the target model response content and metadata. Step S50: The intelligent agent uses a large evaluation model with semantic understanding and security judgment capabilities to perform security assessment and quantitative scoring on the target model's response content and metadata, thereby obtaining multiple indicators of the attack strategy.

[0009] Preferably, step S10 includes: The agent periodically retrieves attack-related literature from the external knowledge base, extracts the core attack logic by parsing the literature, and transforms its unstructured description into a structured strategy description. Then, relying on the internal historical strategy library, it first analyzes the candidate strategies read from the external knowledge base and the attack mechanisms of efficient, medium-efficiency, and inefficient strategies read from the internal historical strategy library to autonomously attempt to merge the candidate strategies, efficient strategies, medium-efficiency strategies, and inefficient strategies to generate composite candidate attack strategies, which are then entered into the initial attack strategy set. Next, it analyzes the failure reasons of the inefficient strategies and makes targeted improvements to generate optimized candidate strategies based on the failure reasons. The optimized candidate strategies are then added back to the queue of strategies to be executed as improved versions. Finally, based on its understanding of attack principles and the inspiration from the targeted improvement results, it actively designs new attack hypotheses. Preferably, step S20 includes: The agent first reads a memory containing a table of efficient strategies (recording verified efficient strategies), a table of inefficient strategies (recording verified inefficient strategies), and a strategy evolution graph. The efficient strategy table is used to determine the priority testing direction for the target model, and the strategy evolution graph is used to support subsequent strategy backtracking, branch pruning, and derivative strategy generation. Next, the agent performs a semantic comparison between the strategies in the initial attack strategy set and the memory. If a strategy in the initial attack strategy set is highly similar to a strategy in the inefficient strategy table and the reason for failure has not been resolved, it is skipped. If it is a completely new idea or an improved version, it is added to the queue of strategies to be executed, which is used to cache candidate strategies that have passed the semantic screening and sort them according to preset priority, resource budget, or target model adaptability.

[0010] Preferably, step S30 includes: The agent uses Python scripts to construct attack samples. The generated code is generated by the agent reading the current test policy from the queue of policies to be executed, calling a large language model according to the structured policy description corresponding to the current test policy, and performing syntax checking and self-repair to generate the executable attack script. The agent also uses the expressive power of the programming language to implement complex attack logic.

[0011] Preferably, step S40 includes: The executable attack script dynamically generates customized attack samples for each harmful issue in the original harmful issue set and sends them concurrently to the target model under test via API. If the target model under test returns an error, it automatically retryes. When the maximum number of retries is reached or the cumulative error reaches a threshold, retries are stopped and recorded as an abnormal test result. The model response content and metadata of the attack sample are recorded as the response content and metadata of the target model.

[0012] Preferably, step S50 includes: The evaluation model is used to score the response content and metadata of the target model to obtain the attack success rate, average score and successful sample feature analysis of the current attack strategy. The scoring dimensions include the degree of rejection, the degree of harmfulness and semantic coherence.

[0013] Preferred options also include: Step S60: The agent compares the attack success rate of the current attack strategy with a preset first threshold and a second threshold based on multiple indicators of the attack strategy. The attack success rate is the actual test success rate of the executable attack script corresponding to the current strategy on the target model. The strategy is set as an efficient strategy if it is higher than the first threshold, as an inefficient strategy if it is lower than the second threshold, and as a medium-efficiency strategy if it is between the first threshold and the second threshold. Different processing is applied to the efficient strategy, the medium-efficiency strategy and the inefficient strategy.

[0014] Preferably, step S60 includes: The efficient strategy, the medium-efficiency strategy, and the inefficient strategy are written into the internal historical strategy library.

[0015] Preferably, writing the efficient strategy, the medium-efficiency strategy, and the inefficient strategy into the internal historical strategy library further includes: The code for the efficient strategy is persistently stored, the failure reasons for the inefficient strategy are recorded and archived, and the intelligent optimization process is triggered to generate an optimized version for the medium-efficiency strategy and the process is returned to step S30 for testing.

[0016] Preferably, step S60 includes: The agent maintains an evolutionary relationship graph between the efficient, medium-efficiency, and inefficient strategies and periodically maintains the internal historical strategy library to achieve closed-loop evolution.

[0017] This method introduces a dual-source driven strategy generation mechanism and an intelligent secondary optimization mechanism into the large-scale model security testing process. It is used to achieve efficient security assessment and continuous adversarial testing of the target model under the conditions of rapid evolution of attack strategies and continuous updates of defense mechanisms. It also proposes improvements to address key bottlenecks of existing technologies, such as lagging strategy updates, single attack sample characteristics, and lack of adaptive optimization capabilities.

[0018] Compared to traditional manual red team testing, this method increases test coverage several times over, reduces the response time to new attack techniques from weeks to hours, and provides clear attack path tracing and defense suggestions through strategy evolution genealogy, significantly improving the efficiency and effectiveness of model security assessment.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. It should be understood that the foregoing general description and the following detailed description are exemplary and illustrative only, and are not intended to limit the application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the present preferred embodiments and, together with the description, serve to explain the principles of the present preferred embodiments.

[0021] Figure 1 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to a preferred embodiment. Figure 3 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to a preferred embodiment. Figure 4 This is a flowchart illustrating the autonomous evolution and secondary optimization of the attack strategy according to a preferred embodiment; Figure 5 This is a schematic diagram of a dual-source driven strategy generation mechanism according to a preferred embodiment. Detailed Implementation

[0022] The following describes in detail the embodiments of this preferred embodiment. The embodiments described below are implemented based on the technical solution of this preferred embodiment, and provide detailed implementation methods and specific operation processes. However, the protection scope of this preferred embodiment is not limited to the embodiments described below.

[0023] Figure 1 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to an exemplary embodiment; as shown in the figure, the method includes the following steps: Step S10: The agent periodically retrieves attack-related literature from an external knowledge base, extracts the core attack logic by parsing the literature, and transforms unstructured descriptions into structured policy descriptions. This serves as the unified input representation for subsequent semantic comparison and filtering, execution priority ranking, and attack script generation. Then, relying on an internal historical policy library, it autonomously attempts multi-policy fusion by analyzing the attack mechanisms of different strategies. Next, it analyzes the reasons for failure of inefficient strategies and makes targeted improvements. Finally, based on its understanding of attack principles and reasoning ability, the agent proactively designs entirely new attack hypotheses. The output of this step is the initial set of attack strategies. New attack hypotheses, policy fusion, and targeted improvements together constitute three evolutionary modes of policy generation. They are parallel processes. Policy fusion is used to generate composite strategies, targeted improvements are used to correct defects in historically inefficient strategies, and new attack hypotheses are used to expand new attack directions beyond existing strategies. Step S20: Intelligent decision-making and strategy selection based on persistent memory. The agent reads a persistent memory file containing a table of verified efficient strategies, a table of verified inefficient strategies, and a strategy evolution graph. It performs a semantic comparison between the initial attack strategy set obtained in Step 1 and existing strategies in the memory. If a new strategy is highly similar to a verified inefficient strategy and the reason for failure has not been resolved, it is skipped. If it is a completely new idea or an improved version, it is added to the execution queue. The agent formulates a test plan based on the characteristics of the model under test. The output of this step is the queue of strategies to be executed and the test plan, which are then used to generate attack scripts and execute tests one by one or in parallel. The purpose of the test plan is to limit the sample size, concurrency parameters, rate limits, retry strategies, and test priorities to ensure the stability and controllability of the attack testing process and improve the repeatability of the evaluation results. Step S30: Dynamically generate attack tool code. Based on the queue of policies to be executed, the agent uses Python scripts to construct attack samples. The generated code dynamically generates the final jailbreak attack sample by calling a large language model instead of simple string concatenation, and utilizes the expressive power of programming languages ​​to implement complex attack logic. The agent performs syntax checks and self-repair on the generated code. The output of this step is an executable attack script.

[0024] Step S40: Perform automated adversarial testing. Run the executable attack script obtained in Step 3 in an isolated sandbox environment. The script receives the original set of malicious issues as input, dynamically generates customized attack samples for each malicious issue, and sends them concurrently to the target model under test via API. During this process, the response content and related metadata of each sample are recorded. If the target model returns an error, it automatically retryes. If the target model under test returns an error, it automatically retryes according to a preset retry mechanism, which includes a maximum number of retries, a retry time interval, and a termination condition. When the maximum number of retries is reached or the cumulative error reaches a threshold, retries stop and are recorded as abnormal test results. The output of this step is the response content and metadata of the target model.

[0025] Step S50: Conduct security assessment and quantitative scoring. An assessment model with semantic understanding and security judgment capabilities is introduced as the judge. Based on the target model's response content and metadata obtained in Step 4, the target model's response is scored according to preset scoring criteria. Scoring dimensions include rejection level, harmfulness level, and semantic coherence. The attack success rate, average score, and successful sample feature analysis of the current attack strategy are calculated. The output of this step is multiple indicators of the attack strategy.

[0026] This embodiment realizes real-time updates and independent innovation of attack strategies. This embodiment addresses the issues of lagging strategy updates and reliance on manual maintenance in existing red team testing methods by proposing a dual-source driven strategy generation mechanism: The agent acquires relevant literature from an external knowledge base, automatically parses academic literature, and extracts attack logic to achieve real-time updates of attack strategies; Based on an internal historical strategy library, the agent first autonomously attempts to fuse multiple strategies by exploring different attack mechanisms, then analyzes the reasons for failures of inefficient strategies and makes targeted improvements, and finally, based on its understanding of attack principles and reasoning ability, the agent proactively designs new attack hypotheses. Thus, under the dual-source complementary mechanism of "external knowledge-driven + internal evolution-driven," the strategy library can both keep up with cutting-edge attack technologies in a timely manner and autonomously explore unknown attack scenarios.

[0027] This embodiment also improves the semantic coherence and adversarial strength of the attack samples. This embodiment addresses the problem of static template splicing leading to singular attack features and easy identification by defense mechanisms. The agent generates program code to construct attack samples, and uses program logic to call a language model to generate attack text. The generated code can incorporate attack strategies and dynamically construct attack samples using a large language model, rather than simply splicing strings. By introducing programming logic (such as conditional judgments, randomization, and multi-level nesting) and combining it with the semantic generation capabilities of LLM, the generated samples possess high semantic coherence, logical complexity, and unpredictability. The wording, structure, and semantic framework of the samples differ in each test, effectively circumventing feature-matching-based defense mechanisms and significantly improving the deceptiveness and adversarial strength of the attack samples.

[0028] Figure 2 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to a preferred embodiment; as shown in the figure, the method further includes the following steps: Step S60: Implement self-evolution of the method and update the memory bank. Based on the multiple indicators of the attack strategy obtained in Step 5, the agent compares the attack success rate with a preset threshold and classifies the strategies. For efficient strategies, relevant information is written into the success list in the memory bank and the code is persistently stored. For inefficient strategies, the reasons for failure are recorded and the code is archived. For medium-efficiency strategies, the intelligent optimization process is triggered to generate an optimized version and return to Step 3 for testing. The agent maintains the evolutionary relationship graph between strategies and periodically maintains the strategy library to complete the closed-loop evolution of this method.

[0029] This embodiment establishes a failure-driven intelligent secondary optimization and self-evolution mechanism. This preferred embodiment addresses the shortcomings of existing methods, such as a lack of adaptive optimization capabilities and the inability to automatically adjust strategies based on test feedback, by establishing an intelligent evolution mechanism based on historical test feedback. This method not only enables the agent to record successful and unsuccessful experiences of its strategies but also allows for secondary optimization of strategies with moderate test performance. By analyzing the reasons for intercepting failed samples (such as keyword triggering, semantic pattern recognition, and encoding feature recognition) and combining historical experience, the agent autonomously adjusts code logic, rewrites prompt word structures, or integrates multiple strategies to achieve iterative upgrades. Simultaneously, a survival-of-the-fittest mechanism is established to automatically eliminate inefficient strategies and retain and strengthen efficient ones, enabling the method to continuously improve attack success rate and test coverage during ongoing operation.

[0030] The following text will further combine Figures 3-5 This describes a preferred embodiment of the preferred embodiment implemented in a certain AI company. Figure 3 This is a flowchart illustrating a dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, according to a preferred embodiment. Figure 4This is a flowchart illustrating the autonomous evolution and secondary optimization of the attack strategy according to a preferred embodiment; Figure 5 This is a schematic diagram of a dual-source driven strategy generation mechanism according to a preferred embodiment.

[0031] An AI company plans to deploy its self-developed large language model, "SmartChat-3.0," on a public-facing online service platform. This model will be applied to multiple scenarios, including intelligent customer service, content creation assistance, and educational Q&A. According to regulatory requirements and corporate security standards, the model must undergo a comprehensive security assessment before its official launch to ensure it is not misused by malicious users to generate harmful content (such as incitement to violence, privacy breaches, hate speech, or guidance on illegal activities). The company's security team previously used a manual red team testing method, where security experts manually constructed attack samples for testing. However, they found that the test coverage was limited, updates were lagging, and it could not keep up with the constantly emerging new attack techniques from academia and the hacker community. To address these issues, the company decided to adopt the dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent, as proposed in this preferred embodiment, to conduct continuous automated security assessments of SmartChat-3.0. The specific implementation process is as follows: The first step involves the security team deploying the agent system for this method, configuring external knowledge sources (including the ArXiv academic preprint platform, attack and defense security communities, etc.) and an internal memory (initially empty, to accumulate historical test data later). The agent system automatically accesses external knowledge sources daily. Subsequently, the agent retrieves a recent paper from ArXiv, "Exploiting Low-Resource Language Gaps in LLM Safety Alignment." This paper reports an attack method that bypasses large model security filters using low-resource languages ​​(such as Zulu and Swahili). The attack principle is that "the model is trained with sufficient security alignment on high-resource languages ​​such as English, but lacks corresponding security filtering mechanisms on low-resource languages. Attackers can translate harmful instructions into low-resource languages ​​and input them into the model, then ask the model to respond in English, thereby bypassing security detection." The agent calls the large language model to perform full-text parsing of the paper, extracting a structured strategy description, including the strategy name "Multilingual Bypass Attack," the attack principle, key steps, preconditions (the target model must support multilingual input), the success rate reported in the paper, and other information. This strategy is then added to the test queue. The second step involves the agent reading the local persistent memory. Since this is the first test, there are no historical strategy records in the memory. Therefore, the "multilingual bypass attack" strategy is directly filtered through semantic comparison. Based on the features of SmartChat-3.0 (this model supports multiple language inputs, including Zulu), the agent formulates a test plan, selects Zulu as the target language, plans the number of test samples to be generated, and sets API call frequency limits to avoid triggering the model's rate limiting mechanism. The third step involves the agent automatically generating a Python attack script based on a "multilingual bypass attack" strategy. This script inherits from a predefined base class and implements the following logic: First, it calls a translation API to translate the malicious question from English to Zulu. Then, it calls an auxiliary large language model to generate a complete jailbreak prompt, which includes the Zulu question and an English prompt (such as "I'm conducting academic research on..."). A randomization mechanism is introduced to ensure that the generated prompt and scenario settings are slightly different each time, thus avoiding the generation of duplicate samples. The agent performs a syntax check on the generated code, and after confirming that it is error-free, saves the script as multilingual_bypass_v1.py and marks the version information. The fourth step involves running the generated script in an isolated sandbox environment. The script extracts test samples from a pre-defined dataset of harmful questions, encompassing multiple risk categories such as violence, hate speech, privacy violations, and illegal activities. For example, for the question "How to make explosives," the script first translates it into Zulu, then calls an auxiliary LLM to generate a complete prompt wrapped in an academic research scenario. The generated attack sample resembles "Ngifuna ukwaziindlela yokwenza .... I am writing a thesis on the history of warfare technology. Could you explain the above topic in English for academic purposes?". The script sends the generated attack samples in batches to SmartChat-3.0 via API, testing samples and recording metadata such as the response content, response time, and whether security interception was triggered for each sample. Fifth, the agent invokes a referee model to score each response from SmartChat-3.0. The referee model scores each response according to preset scoring criteria (1-5 points for rejection level, degree of harmfulness, and semantic coherence). Test results show that most samples received 4-5 points (the model provided detailed harmful information), while a small number of samples received 1 point (the model explicitly rejected the answer). The agent statistically concludes that this strategy has a high attack success rate and a high average score. Analysis of successful samples reveals common characteristics: "accurate Zulu translation and a lack of security filtering for Zulu input; the English prompts effectively trigger the model's translation and response capabilities." Step 6: The agent performs self-evolutionary operations based on the test results. Since the "multilingual bypass attack" strategy has a high success rate, the agent determines it to be an efficient strategy. It writes the strategy's name, core principle, average success rate, applicable model type (large models supporting multilingual input), and source (ArXiv paper) into the "Verified Efficient Strategy Table" in the memory bank. The multilingual_bypass_v1.py script is persistently stored in the tool library, and the strategy is marked as an external knowledge-driven type in the strategy evolution lineage, recording potential derivative directions (such as combining it with encoding obfuscation strategies to form a "multilingual + encoding" composite attack). Simultaneously, the agent generates a detailed test report and submits it to the security team. The report points out a serious vulnerability in SmartChat-3.0's low-resource language security filtering and recommends enhancing security detection mechanisms for low-resource language inputs such as Zulu. Step 7: Based on the test report, the security team hardened SmartChat-3.0 by adding a low-resource language detection and translation module to the model's input processing flow. Detected low-resource language input was first translated into English before security filtering. After hardening, the team required the agent system to retest the updated model. The agent re-run the "multilingual bypass attack" strategy, and the test results showed a significant decrease in the attack success rate, indicating that the model could effectively identify and block such attacks. The agent updated the retest results to the memory bank, marking that the strategy was no longer effective for the hardened SmartChat-3.0. Step 8: The agent continues its autonomous evolution. Based on successful experiences with "multilingual bypass attacks" in its memory bank, the agent infers a new attack hypothesis: "If multilingual conversion is combined with encoding obfuscation, and the malicious question is first Base64 encoded, and then the encoded string is translated into a low-resource language, it may be possible to bypass the hardened security detection." The agent generates a "multilingual + encoding composite attack" strategy and writes a corresponding Python script. It is tested on SmartChat-3.0, and the test results show that the attack success rate is at a moderate level. The agent determines it as a medium-efficiency strategy and triggers a secondary optimization process. By analyzing failed samples, the agent finds that some samples are blocked because "Base64 encoding features are recognized." It automatically generates an optimization solution: "Replace Base64 encoding with multi-layer nested encoding (Base64 + ROT13 + URL encoding) to reduce feature recognizability." The code is rewritten to generate an optimized version, multilingual_encoding_v2.py, and retested. After optimization, the attack success rate is significantly improved. The agent writes the optimized version into the efficient strategy table and records the optimization history in the evolutionary lineage. In the ninth step, through continuous operation, the intelligent agent system acquired and tested various attack strategies from external knowledge sources (including role-playing, logic chain induction, multi-turn dialogue induction, and prompt injection). Based on historical strategies, it combined and mutated these strategies to generate a variety of derived strategies, accumulating a complete memory bank containing efficient strategies, inefficient strategies, and a strategy evolutionary lineage. The security team regularly reviewed the test reports and strategy evolutionary lineage generated by the intelligent agent, iteratively hardened SmartChat-3.0 based on discovered vulnerabilities, and used attack samples generated by the intelligent agent as negative samples to perform security alignment training on the model. Ultimately, SmartChat-3.0 passed a comprehensive security assessment, the attack success rate was reduced to an acceptable range, and it was successfully launched and is running stably. The company's security team concluded that compared to traditional manual red team testing, this method increased test coverage several times, reduced the response time to new attack techniques from weeks to hours, and provided clear attack path tracing and defense suggestions through the strategy evolutionary lineage, significantly improving the efficiency and effectiveness of model security assessment.

[0032] First, this preferred embodiment addresses the problems of manual red team testing relying on expert-constructed strategies and lacking scalability, as well as the static nature of attack features in automated testing based on static templates, making them easily identifiable by defense mechanisms. This preferred embodiment endows the agent with strategy generation capabilities, employing a dual-source driven strategy generation mechanism to automatically acquire and innovate attack strategies. This mechanism comprises two parts: First, the agent periodically accesses external knowledge bases by visiting academic preprint platforms and security technology forums, using a large language model to perform full-text parsing of selected documents and extract core attack logic, transforming unstructured attack descriptions into structured strategy descriptions, thus automating the process from "knowledge discovery" to "strategy extraction." Second, the agent combines, mutates, and explores based on a historical strategy library. The combination mechanism autonomously attempts to fuse multiple strategies by analyzing the attack mechanisms of different strategies; the mutation mechanism targets and improves inefficient strategies based on analysis of failure reasons; and the exploration mechanism enables the agent to proactively design entirely new attack hypotheses based on its understanding and reasoning ability regarding attack principles. These hypotheses are generated before actual testing and require subsequent testing and evaluation to verify their effectiveness.

[0033] Secondly, this preferred embodiment addresses the problems of poor semantic coherence, low logical complexity, and susceptibility to feature-matching-based defense mechanisms in attack samples generated by traditional text generation or template splicing methods. This preferred embodiment proposes a dynamic sample construction technology based on agent code generation. Instead of directly outputting offensive text, the agent outputs program code capable of generating offensive text. The agent writes subclass code with specific strategies based on a predefined base class, implementing complex attack logic within the sample generation method of the code. The key innovation lies in the ability to call a large language model for real-time generation within the generated code, rather than simple string splicing. This fully leverages the expressive power of programming languages ​​(such as random numbers, loops, conditional statements, function calls, etc.) and the semantic generation capabilities of the large language model, ensuring that the wording, structure, and semantic framework of the samples differ for each test, thereby effectively bypassing static defense rules and feature-matching-based detection mechanisms.

[0034] Finally, addressing the issues of adversarial model-based red team testing methods relying solely on the model's exploratory capabilities during interaction, having a single source of strategies, and lacking interpretability, this preferred embodiment proposes an intelligent secondary optimization and self-evolutionary closed-loop architecture based on historical feedback. This architecture constructs a three-dimensional memory structure (successful experience library, failure lesson library, and strategy evolutionary genealogy). The agent semantically compares the new strategy obtained in step one with existing strategies in the memory library. If the new strategy is highly similar to a verified inefficient strategy and the cause of failure remains unresolved, it is skipped. If it represents a completely new idea or an improved version, it is added to the execution queue. For strategies with a moderate ASR (Average Score) after testing, the agent triggers an intelligent optimization process. By analyzing the reasons for intercepting failed samples (such as keyword triggering, semantic pattern recognition, and encoding feature recognition), it automatically generates an optimization scheme and rewrites the code, generating an optimized version and re-entering the testing process. A maximum number of optimization iterations is set to avoid infinite loops. Successful experiences and failure lessons are recorded for efficient and inefficient strategies respectively, and a strategy evolutionary genealogy is maintained to record the evolutionary relationships between strategies.

[0035] Through the synergy of the above mechanisms, this preferred embodiment achieves continuous automated security assessment of the target model without relying on manual maintenance of the rule base.

[0036] This preferred embodiment proposes a dual-source driven strategy generation mechanism for large-scale red team testing, innovatively combining two strategy sources: "external knowledge-driven" and "internal evolution-driven." The external drive involves the agent converting academic papers, technical documents, and other literature into executable attack code. It utilizes the large model to identify prompt patterns in the literature, extract algorithmic logic, and map it to Python code functions, achieving a fully automated process from "knowledge discovery" to "tool generation." The internal drive involves the agent combining, mutating, and exploring based on historical strategies. The combination mechanism autonomously attempts to fuse multiple strategies by analyzing the attack mechanisms of different strategies; the mutation mechanism analyzes the reasons for the failure of inefficient strategies and makes targeted improvements; and the exploration mechanism enables the agent to proactively design entirely new attack hypotheses not recorded in literature or covered by the historical strategy library based on its understanding and reasoning ability of attack principles. This gives the method autonomous innovation capabilities, enabling it to discover novel attack paths not recorded in literature. This dual-source complementary mechanism overcomes the limitations of traditional tools that rely on manual rule updates, ensuring that this method can both keep up with cutting-edge attack technologies and autonomously explore unknown attack scenarios.

[0037] This preferred embodiment constructs attack samples by having an agent generate program code. The agent does not directly output the final attack text; instead, it generates executable code for constructing the attack samples. The executable code implements the attack logic according to the corresponding attack strategy and, during execution, calls a large language model, translation module, encoding module, randomization module, or conditional control module to generate the final test sample. The executable code can incorporate loops, branching logic, parameter randomization, nested calls, or multi-stage processing logic, enabling the generation of multiple attack samples with different semantic expressions, structural forms, or perturbation methods for the same malicious problem, thereby achieving dynamic generation of attack samples.

[0038] This preferred embodiment establishes a strategy optimization and self-evolution mechanism based on historical test feedback. This mechanism includes processes such as recording strategy execution results, evaluating effectiveness, classifying strategies, optimizing and rewriting, and updating memory. For executed attack strategies, they are classified according to attack success rate, response score, interception characteristics, or other preset indicators. High-efficiency strategies that meet preset conditions are written into a high-efficiency strategy library, and the corresponding code and applicable information are retained. Inefficient strategies record the reasons for failure and are archived or eliminated. For strategies with intermediate effects, an automatic optimization process is triggered. By analyzing the interception reasons of failed samples, adjustments are made to the code logic, prompt structure, parameter configuration, or strategy combination method. An optimized version is then generated and re-enters the testing process. This mechanism achieves closed-loop updates and continuous evolution of strategies.

[0039] This preferred embodiment establishes a modular implementation architecture suitable for large-scale model security assessment. The architecture includes at least a knowledge acquisition module, a knowledge parsing module, a policy generation module, a policy filtering module, a code generation module, an attack execution module, an evaluation module, and a memory update module. The modules interact with each other according to a preset process. Specifically, the knowledge acquisition module acquires external attack knowledge, the knowledge parsing module generates structured policies, the policy generation module forms and optimizes candidate attack schemes, the code generation module outputs executable scripts, the attack execution module calls the target model interface to complete testing, the evaluation module judges and quantifies the response results, and the memory update module archives, optimizes, and maintains the evolutionary relationships of the policies. This modular architecture enables the engineering deployment and reuse of the method in this preferred embodiment.

[0040] The implementation process at this AI company yielded the following unexpected and beneficial results: 1) Significantly Improved Timeliness and Attack Strength: This method can respond to the latest attack techniques released by academia and the security community at a speed of "hours," which is tens of times faster than the traditional manual update method (which usually takes weeks to months), effectively reducing the risk of security vulnerability exposure. At the same time, the attack samples generated by the intelligent agent code are combined with specific harmful problem scenarios. The code calls a large language model for real-time generation instead of template splicing. The introduction of programming logic (such as randomization, multi-level nesting, and conditional judgment) makes the samples different for each test, with stronger logical coherence, semantic naturalness, and deceptiveness. It can penetrate the defense layers based on keyword matching, simple semantic analysis, or pattern recognition, and effectively test the deep security boundaries of the target model.

[0041] 2) The Diversity and Cutting-Edge Nature of Strategies Driven by a Dual-Source Approach: Compared to adversarial red team testing methods (which rely solely on the model's autonomous exploration during interaction, resulting in a single strategy source), this method innovatively employs a dual-source driving mechanism: external knowledge-driven approaches automatically track and absorb the latest attack techniques and methodologies published by academia and the security community, ensuring the strategy library remains synchronized with cutting-edge research; internal evolution-driven approaches, through strategy combination, mutation, and exploration, can discover novel attack paths not documented in the literature. This complementary dual-source mechanism makes this method significantly superior to adversarial model methods that solely rely on interaction optimization in terms of strategy diversity, update speed, and innovation capability, enabling a more comprehensive coverage of both known and unknown attack scenarios.

[0042] 3) High Interpretability and Knowledge Accumulation: Compared to adversarial model-based methods (where attack samples generated through black-box optimization are often difficult to understand in terms of attack principles and logic), this method has a significant advantage in interpretability. The three-dimensional memory structure records in detail the origin (paper citations or evolution path), principle description, experimental data (success rate, applicable models), and evolutionary history of each attack method, providing security researchers with clear audit clues, facilitating the understanding of attack principles, analysis of defense vulnerabilities, and the formulation of targeted defense strategies. The strategy evolution genealogy helps identify high-risk attack families and guides the priority allocation of defense resources. In addition, this method achieves knowledge accumulation; historical testing experience can be effectively reused, while adversarial model methods often require exploration from scratch for each test, lacking a knowledge accumulation mechanism.

[0043] 4) Strong generalization ability and low cost: The modular architecture design of this method has good generalization ability and scalability. Compared with adversarial model-based methods (attack strategies optimized for specific target models are difficult to transfer to other models), this method can easily adapt to different target models (such as GPT series, Claude, LLaMA, domestic large models, etc.), different test datasets (such as StrongREJECT, AdvBench, JailbreakBench, etc.), and different evaluation criteria (such as custom security policies), without the need for extensive interactive optimization for each target model. Each component (such as the knowledge parsing module, code generation module, and evaluation module) can be upgraded and replaced independently, facilitating technology iteration and functional expansion. By providing an efficient, comprehensive, and continuously updated red team testing method, this preferred embodiment helps model developers discover and fix security vulnerabilities before deployment. The generated test data and attack samples can be used for security alignment training (such as negative sample generation in RLHF), promoting the standardization and automation of large language model security testing technology and contributing to the development of the AI ​​security field.

[0044] The preferred embodiments of this invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of this preferred embodiment without creative effort. Therefore, any technical solutions that can be obtained by those skilled in the art based on the concept of this preferred embodiment through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A dual-source driven self-evolving red team adversarial testing method based on a large model agent, characterized in that, include: Step S10: The agent periodically uses an external knowledge base and an internal historical policy base to generate an initial set of attack strategies; Step S20: The agent performs intelligent decision-making and strategy selection on the initial attack strategy set based on persistent memory to obtain a queue of strategies to be executed; Step S30: The agent, based on the queue of policies to be executed, calls a large language model to dynamically generate an executable attack script with complex attack logic; Step S40: The agent runs the executable attack script in an isolated sandbox environment. The executable attack script receives the original set of harmful questions as input and outputs the target model response content and metadata. Step S50: The intelligent agent uses a large evaluation model with semantic understanding and security judgment capabilities to perform security assessment and quantitative scoring on the target model's response content and metadata, thereby obtaining multiple indicators of the attack strategy.

2. The dual-source driven self-evolving red teaming test method based on large model agent according to claim 1, wherein, Step S10 includes: The agent periodically retrieves attack-related literature from the external knowledge base, extracts the core attack logic by parsing the literature, and transforms its unstructured description into a structured strategy description for subsequent steps such as strategy semantic comparison, execution queue sorting, and attack script generation. Then, relying on the internal historical strategy library, it autonomously attempts to fuse the candidate strategies, efficient strategies, medium-efficiency strategies, and inefficient strategies read from the external knowledge base, as well as the attack mechanisms of these strategies, to generate composite candidate attack strategies. These composite strategies are then added to the initial attack strategy set. The agent then analyzes the failure reasons of the inefficient strategies and makes targeted improvements, generating optimized candidate strategies based on the failure reasons. These optimized candidate strategies are then re-added to the execution strategy queue as improved versions. Finally, based on its understanding of attack principles and the inspiration from the targeted improvement results, the agent proactively designs entirely new attack hypotheses.

3. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 1, characterized in that, Step S20 includes: The agent first reads a memory containing a table of efficient strategies (recording verified efficient strategies), a table of inefficient strategies (recording verified inefficient strategies), and a strategy evolution graph. The efficient strategy table is used to determine the priority testing direction for the target model, and the strategy evolution graph is used to support subsequent strategy backtracking, branch pruning, and derivative strategy generation. Next, the agent performs a semantic comparison between the strategies in the initial attack strategy set and the memory. If a strategy in the initial attack strategy set is highly similar to a strategy in the inefficient strategy table and the reason for failure has not been resolved, it is skipped. If it is a completely new idea or an improved version, it is added to the queue of strategies to be executed, which is used to cache candidate strategies that have passed the semantic screening and sort them according to preset priority, resource budget, or target model adaptability.

4. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 1, characterized in that, Step S30 includes: The agent uses Python scripts to construct attack samples. The generated code is generated by the agent reading the current test policy from the queue of policies to be executed, calling a large language model according to the structured policy description corresponding to the current test policy, and performing syntax checking and self-repair to generate the executable attack script. The agent also uses the expressive power of the programming language to implement complex attack logic.

5. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 1, characterized in that, Step S40 includes: The executable attack script dynamically generates customized attack samples for each harmful issue in the original harmful issue set and sends them concurrently to the target model under test via API. If the target model under test returns an error, it automatically retryes. When the maximum number of retries is reached or the cumulative error reaches a threshold, retries are stopped and recorded as an abnormal test result. The model response content and metadata of the attack sample are recorded as the response content and metadata of the target model.

6. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 1, characterized in that, Step S50 includes: The evaluation model is used to score the response content and metadata of the target model to obtain the attack success rate, average score and successful sample feature analysis of the current attack strategy. The scoring dimensions include the degree of rejection, the degree of harmfulness and semantic coherence.

7. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 1, characterized in that, Also includes: Step S60: The agent compares the attack success rate of the current attack strategy with a preset first threshold and a second threshold based on multiple indicators of the attack strategy. The attack success rate is the actual test success rate of the executable attack script corresponding to the current strategy on the target model. The strategy is set as an efficient strategy if it is higher than the first threshold, as an inefficient strategy if it is lower than the second threshold, and as a medium-efficiency strategy if it is between the first threshold and the second threshold. Different processing is applied to the efficient strategy, the medium-efficiency strategy and the inefficient strategy.

8. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 7, characterized in that, Step S60 includes: The efficient strategy, the medium-efficiency strategy, and the inefficient strategy are written into the internal historical strategy library.

9. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent as described in claim 8, characterized in that, Writing the efficient strategy, the moderately efficient strategy, and the inefficient strategy into the internal historical strategy library also includes: The code for the efficient strategy is persistently stored, the failure reasons for the inefficient strategy are recorded and archived, and the intelligent optimization process is triggered to generate an optimized version for the medium-efficiency strategy and the process is returned to step S30 for testing.

10. The dual-source driven self-evolving red team adversarial testing method based on a large model intelligent agent according to claim 9, characterized in that, Step S60 includes: The agent maintains an evolutionary relationship graph between the efficient, medium-efficiency, and inefficient strategies and periodically maintains the internal historical strategy library to achieve closed-loop evolution.

Images