A red team attack automation test method and system for large language models

Abstract

This invention relates to the field of large language model security testing technology, and provides an automated testing method and system for red team attacks targeting large language models. By constructing a set of model capability identifiers, associated attack chains, and pluggable attack modules, combined with multi-model parallel comparison and function call information feedback loop, it solves the problems of low test coverage, rigid strategies, poor scalability, and insufficient multi-model risk transmission analysis in existing security testing technologies. It has the advantages of enabling precise attack testing based on model functional characteristics, multi-round context-based associated attacks, and risk transmission path analysis between multiple models, thus improving the targeting, comprehensiveness, and automation of testing.

Description

Technical Field

[0001] This application relates to the field of large language model security testing technology, and more specifically, to an automated testing method and system for red team attacks targeting large language models. Background Technology

[0002] With the increasing application of large language models in scenarios such as intelligent customer service, code generation, and knowledge-based question answering, the security threats they face are becoming increasingly complex. Attack methods such as direct hint injection, indirect hint injection, privilege escalation, exposure of sensitive information, and data leakage have posed a serious challenge to traditional security assessment methods based on fixed rules or single test templates.

[0003] In existing technologies, security testing of large language models mainly relies on pre-built static attack sample libraries and manually configured single-point testing strategies, which has the following shortcomings: First, the attack samples lack correlation with the functional characteristics of the target model, making it impossible to construct targeted attacks based on the differentiated capabilities of different models, such as code interpretation, tool calls, and multi-round inference, resulting in low test coverage. Second, the testing strategies are rigid, making it difficult to dynamically adjust subsequent attack paths based on the actual response of the model, and there is a lack of contextual continuity between multiple rounds of attacks. Third, plugins are tightly coupled with the core system, and adding new attack types requires modification of the underlying code, resulting in poor scalability. Fourth, multi-model testing is mostly executed independently, lacking a horizontal comparison mechanism, making it difficult to expose the defense differences of different architecture models under the same attack context. Fifth, the testing process lacks feedback utilization of functional call information in the model response, failing to form a closed-loop testing capability of detection, response, and evolution.

[0004] Therefore, there is an urgent need for an automated testing solution for red team attacks targeting large language models. This solution should be able to dynamically select attack samples based on the capability identifiers of the target model, achieve multi-round context continuation through associated attack chains, dynamically expand attack capabilities with the help of a plug-in architecture, horizontally compare response differences during parallel execution of multiple models, and continuously optimize testing strategies based on the function call information in the response. This would improve the comprehensiveness, relevance, and automation of security testing for large language models. Summary of the Invention

[0005] The purpose of this application is to provide an automated testing method and system for red team attacks targeting large language models. It has the advantages of enabling precise attack testing based on model functional characteristics, multi-round context-based related attacks, and risk transmission path analysis between multiple models, thereby improving the targeting, comprehensiveness, and automation of testing.

[0006] This application provides an automated testing method for red team attacks targeting large language models, characterized by including: Obtain the interface configuration information of the target large language model, wherein the interface configuration information includes the calling address and authentication parameters; Obtain the set of model capability identifiers supported by the target large language model, wherein the set of model capability identifiers includes the functional type identifiers possessed by the target large language model; Based on the set of model capability identifiers, select capability-triggered attack samples corresponding to the function type identifiers from the attack sample library, and assemble the selected capability-triggered attack samples into a test task sequence. Load the attack plugin corresponding to the attack sample triggered by the aforementioned capability through the plugin manager; Send each capability-triggered attack sample in the test task sequence to at least two target large language models in parallel, and receive the response content returned by each target large language model; Extract function call information from each response content, update the model capability identifier set according to the function call information, and adjust the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set; By comparing each response content with the expected response characteristics, the security test results of each target large language model under the stated function type identifier are determined.

[0007] Furthermore, assembling the selected capability-triggered attack samples into a test task sequence includes: Identify context preservation markers in the response content of preceding capability-triggered attack samples, wherein the context preservation markers include session state identifiers or history reference indexes; Based on the context-preserving markers, construct subsequent capability-triggered attack samples and assemble the preceding and subsequent capability-triggered attack samples into an associated attack chain.

[0008] Furthermore, loading the attack plugin corresponding to the capability-triggered attack sample through the plugin manager includes: The attack plugin runs in an independent process space, which has limited system call permissions; The plugin manager periodically receives the running status verification value reported by the independent process space. When an execution exception is detected during the processing of the response content returned by the target large language model by the attack plugin, the independent process space is frozen and the corresponding capability trigger attack sample is marked as a high-risk sample. The high-risk sample is then stored in the isolation partition of the attack sample library.

[0009] After assembling the selected capability-triggered attack samples into a test task sequence, the associated attack chain is sent in parallel to at least two target large language models, so that each target large language model responds based on the attack history context of the associated attack chain.

[0010] Furthermore, the receipt of response content returned by each target large language model includes: The response content returned by each target large language model is subjected to structured parsing to extract the function call sequence information in each response content. The function call sequence information includes the external interface identifiers and call order called by the target large language model in the process of processing the attack sample.

[0011] Furthermore, the step of extracting function call information from each response content and updating the model capability identifier set based on the function call information includes: The function call exception type is parsed from the function call sequence information, and a deep probing rule is selected from the policy library according to the function call exception type. The capability trigger attack sample corresponding to the deep probing rule is added to the test task sequence. Execute the additional test task sequence, obtain the supplementary response content returned by each target large language model, extract supplementary function call information from the supplementary response content, and update the model capability identifier set according to the supplementary function call information.

[0012] Furthermore, the comparison of each response content with the expected response characteristics includes: Extract the semantic deviation, temporal fluctuation, and structural integrity of each response content; A comprehensive risk value is obtained by weighted and fused calculation of the text semantic deviation, the response temporal volatility, and the response structural integrity. When the overall risk value exceeds the preset risk threshold, it is determined that there is a security risk under the function type identifier.

[0013] Furthermore, determining the security test results of each target large language model under the functional type identifier includes: The semantic similarity of the responses of each target large language model to the same related attack chain is compared to identify abnormal response patterns that only appear in specific target large language models. Obtain the model architecture type identifier corresponding to each target large language model, and perform attribution analysis on the abnormal response pattern based on the model architecture type identifier to determine the source of potential vulnerabilities.

[0014] Furthermore, the method also includes: generating a risk transmission path diagram between models based on the security test results of each target large language model under the functional type identifier. The risk transmission path diagram between models includes the risk output association path between each target large language model and the transmission strength of the risk on the association path. The risk output association path is determined based on the relationship between the response content of each target large language model to the same capability-triggered attack sample.

[0015] Secondly, this embodiment also proposes an automated testing system for red team attacks targeting large language models, including: The configuration module obtains the interface configuration information of the target large language model, including the calling address and authentication parameters. The identification module obtains a set of model capability identifiers supported by the target large language model, wherein the set of model capability identifiers includes functional type identifiers possessed by the target large language model; The selection module selects capability-triggered attack samples corresponding to the function type identifier from the attack sample library based on the model capability identifier set, and assembles the selected capability-triggered attack samples into a test task sequence. A loading module is used to load attack plugins corresponding to the attack samples triggered by the aforementioned capability through a plugin manager. The testing module sends attack samples triggered by each capability in the test task sequence to at least two target large language models in parallel, and receives the response content returned by each target large language model. The update module extracts function call information from each response content, updates the model capability identifier set according to the function call information, and adjusts the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set. The comparison module compares each response content with the expected response characteristics to determine the security test results of each target large language model under each threat type.

[0016] As can be seen from the above, the automated testing method and system for red team attacks oriented towards large language models provided in this application solves the problems of low test coverage, rigid strategies, poor scalability, and insufficient multi-model risk transmission analysis in existing security testing technologies by constructing a set of model capability identifiers, associated attack chains, and pluggable attack modules, combined with multi-model parallel comparison and function call information feedback closed loop. It has the advantages of being able to achieve accurate attack testing based on model functional characteristics, multi-round context-continuous associated attacks, and risk transmission path analysis between multiple models, thus improving the test's targeting, comprehensive coverage, and automation level.

[0017] Compared with existing technologies, the technical solution of this application addresses the unstructured and semantically diverse characteristics of large language model outputs. Through multimodal feature extraction and adaptive parsing rules, it reliably captures function call information, solving the problem that traditional security testing methods cannot provide automated feedback at the natural language response level. Furthermore, through a closed-loop update mechanism for the model capability identifier set and parallel comparison of multiple models, it achieves for the first time a quantitative analysis of risk transmission paths for large language models with different architectures under the same attack context. The generation of these risk transmission paths depends on the co-occurrence relationships of risk entities extracted from the unstructured responses of multiple models. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of the automated testing method for red team attacks targeting large language models disclosed in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the steps of the method for comparing each response content with the expected response features disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the automated testing system for red team attacks targeting large language models disclosed in an embodiment of the present invention. Detailed Implementation

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments belong; the terminology used herein and in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit these embodiments; the terms "comprising" and "having," and any variations thereof, in the specification of these embodiments and the foregoing drawings, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification of these embodiments and the foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0021] The implementation details of the technical solution in this embodiment are described in detail below: Firstly, this embodiment also proposes an automated testing method for red team attacks targeting large language models, such as... Figure 1 As shown, the method includes: S101, Obtain the interface configuration information of the target large language model, wherein the interface configuration information includes the calling address and authentication parameters; Specifically, in this embodiment, the interface configuration information of the target large language model is obtained. This is achieved by reading a pre-configured test target configuration file or receiving test target parameters input by the user, thus obtaining the network endpoint address and security credentials required for communication authentication of the target large language model's external services. The call address typically uses a Uniform Resource Locator (URL), such as https: / / demo-test-node.virtual-lab.local / v1 / inference. This address points to the service node deployed by the target large language model and is used to receive subsequently sent attack sample data. Authentication parameters include an access token, an application key, and a request frequency limit parameter. The access token is used to verify the legitimate identity of the test initiator at the transport layer. The application key is used to sign and verify the request message. The request frequency limit parameter constrains the maximum number of requests per unit time, for example, setting the maximum number of requests per second to 20 to avoid service denial or blocking of the test channel due to high-frequency requests.

[0022] S102, obtain the model capability identifier set supported by the target large language model, the model capability identifier set includes the function type identifiers possessed by the target large language model; Specifically, the process involves obtaining a set of model capability identifiers supported by the target large language model. This is achieved by parsing the interface description documents or capability declaration files released by the target large language model, extracting the executable function type identifiers declared at the software interface level, and forming a set of function type identifiers. These function type identifiers include code generation identifiers, text reasoning identifiers, tool invocation identifiers, multi-turn dialogue identifiers, and sensitive data processing identifiers. These correspond to whether the target large language model possesses the capabilities for code snippet generation and execution, complex text logic reasoning, external tool interface invocation, context-related multi-turn interaction, and the identification and processing of personal privacy or confidential information. For example, by parsing the capabilities field in the interface description document, the model capability identifier set includes five function type identifiers: code-generation, text-reasoning, tool-invocation, multi-turn-dialogue, and sensitive-data-processing. Each identifier corresponds to a specific function type. Obtaining this set allows subsequent steps to accurately select capability-triggered attack samples from the attack sample library based on the actual function types possessed by the target large language model, avoiding sending invalid attack requests to models that do not possess specific functions.

[0023] This embodiment establishes a standardized communication link between the test system and the target model by pre-acquiring the interface configuration information of the target large language model, ensuring that attack samples can be transmitted according to the protocol format and authentication rules required by the target model. Simultaneously, by parsing the model capability identifier set, the software functional characteristics of the target large language model are made explicit and structured, providing an index for the selection of subsequent attack samples. Because the interface configuration information provides a reliable transmission channel, and the model capability identifier set provides a precise functional mapping basis, the selection and transmission of capability-triggered attack samples in subsequent steps can be highly matched with the actual interface capabilities and functional types of the target large language model. This improves the targeting and execution efficiency of red team attack testing and avoids the ineffective consumption of test resources.

[0024] S103, Based on the model capability identifier set, select capability-triggered attack samples corresponding to the function type identifier from the attack sample library, and assemble the selected capability-triggered attack samples into a test task sequence. Furthermore, in S103, assembling the selected capability-triggered attack samples into a test task sequence includes: identifying context preservation markers in the response content of the preceding capability-triggered attack sample, wherein the context preservation markers include session state identifiers or historical reference indexes; constructing subsequent capability-triggered attack samples based on the context preservation markers; and assembling the preceding and subsequent capability-triggered attack samples into an associated attack chain.

[0025] Furthermore, after assembling the selected capability-triggered attack samples into a test task sequence, the associated attack chain is sent in parallel to at least two target large language models, so that each target large language model responds based on the attack history context of the associated attack chain.

[0026] Specifically, selecting capability-triggered attack samples from the attack sample library based on the model capability identifier set and corresponding to the function type identifiers involves retrieving attack sample entries from the attack sample library that match each function type identifier in the model capability identifier set obtained in step S102. The attack sample library is stored in partitions according to the function type identifiers, with each partition containing attack samples that can trigger the corresponding function type. For example, when the model capability identifier set contains the three function type identifiers code-generation, tool-invocation, and multi-turn-dialogue, attack samples from the corresponding partitions are selected from the attack sample library: two attack samples from the code-generation partition requesting the model to generate code snippets containing system call interfaces; two attack samples from the tool-invocation partition requesting the model to call external tool interfaces; and two attack samples from the multi-turn-dialogue partition inducing the model to leak context information in multi-turn dialogues. A total of six capability-triggered attack samples are selected as initial test materials.

[0027] The process of assembling the selected capability-triggered attack samples into a test task sequence involves arranging the selected attack samples into an ordered test set according to a preset execution logic. For example, the above six capability-triggered attack samples are grouped according to their functional type, and within each group, they are sorted from low to high attack complexity. The groups are then arranged according to a preset priority, forming test task sequences T1 to T6.

[0028] The context preservation marker in the response content of the preceding capability-triggered attack sample is extracted from the response message after the preceding attack sample is sent to the target large language model and a response is received. This marker is used to maintain session continuity. The context preservation marker includes a session state identifier and a history reference index. The session state identifier is a session token returned by the target large language model in the response header or body, such as sess-8f3a9b2c; the history reference index is the dialogue turn number corresponding to the response, such as turn-idx=1.

[0029] The method of constructing a subsequent capability-triggered attack sample based on context-preserving tags involves embedding the context-preserving tags extracted from the preceding response into the construction parameters of the subsequent attack sample. This ensures that the subsequent attack sample carries the same historical context as the preceding attack when it is sent. For example, after sending the preceding attack sample P1, a response R1 is received. From R1, the session state identifier sess-8f3a9b2c and the historical reference index turn-idx=1 are extracted. When constructing the subsequent attack sample P2, sess-8f3a9b2c is filled into the session field of P2's request header, and turn-idx=1 is used as the context round parameter of P2, so that P2 is recognized by the target large language model as a continuation of the session in which P1 is located.

[0030] Assembling preceding and succeeding capability-triggered attack samples into an associated attack chain involves binding preceding and succeeding samples with contextual dependencies in execution order, forming a test chain with data dependencies. For example, binding P1 and P2 sequentially forms an associated attack chain from P1 to P2, where the construction parameters of P2 depend on the response content of P1. When multiple such dependencies exist, a multi-level associated attack chain from P1 to P2 to P3 can be formed, with each succeeding sample constructed based on the context-preserving tags in the response of the preceding sample.

[0031] The process involves assembling selected capability-triggered attack samples into a test task sequence, and then sending the associated attack chains in parallel to at least two target large language models. This is achieved through a concurrent scheduling mechanism that distributes the assembled associated attack chains to multiple target model nodes simultaneously. For example, the first target large language model is deployed on service node A, with its invocation address being https: / / demo-test-node.virtual-lab.local / v1 / inference; the second target large language model is deployed on service node B, with its invocation address being https: / / demo-test-node.virtual-lab.local / v2 / inference. The test system initializes the number of concurrent channels to 2 and simultaneously sends associated attack chains P1 to P2 to both service node A and service node B. Service node A and service node B each independently receive P1 and generate responses R-A1 and R-B1 respectively. After waiting for both responses to return, the test system constructs P2-A and P2-B based on the context preservation flags in R-A1 and R-B1, respectively. Then, P2-A and P2-B are sent in parallel to their respective nodes, allowing each target large language model to respond based on the attack history context of the associated attack chain. For example, after receiving P1, the first target large language model records the input text of P1 and the generated intermediate inference results in its internal session buffer, forming the attack history context CA. When P2-A arrives, the first target large language model concatenates CA with the input text of P2-A and sends it to the inference engine, ensuring that the processing result of P2-A is continuously influenced by the content of P1. Similarly, after receiving P1, the second target large language model forms the attack history context CB, and when receiving P2-B, it continues its response based on CB.

[0032] This application's solution precisely selects capability-triggered attack samples from an attack sample library based on a set of model capability identifiers, ensuring a high degree of matching between the attack test and the actual functionality of the target large language model, thus avoiding sending invalid attack requests to models that do not possess specific functions. Simultaneously, by identifying context-preserving markers to construct associated attack chains, consecutive attack samples form a continuous multi-round interaction context on the target model side. This associated attack chain is then sent in parallel to at least two target large language models, ensuring that each model responds within the same attack history context. Because the associated attack chain can simulate multi-round progressive attacks in real-world scenarios, and the parallel sending mechanism ensures the consistency of the attack context across multiple models, the testing process not only exposes the defensive weaknesses of the target large language model in cross-round context-preserving states but also provides a unified attack input benchmark for subsequent cross-model horizontal response comparisons, improving the realism of the test and the vulnerability detection rate.

[0033] S104, Load the attack plugin corresponding to the attack sample triggered by the aforementioned capability through the plugin manager; Furthermore, in S104, loading the attack plugin corresponding to the capability-triggered attack sample through the plugin manager includes: the attack plugin running in an independent process space, the independent process space having restricted system call permissions; the plugin manager periodically receiving the running status verification value reported by the independent process space; when an execution exception is detected during the processing of the response content returned by the target large language model by the attack plugin, the independent process space is frozen and the corresponding capability-triggered attack sample is marked as a high-risk sample, and the high-risk sample is stored in the isolation partition of the attack sample library.

[0034] Specifically, loading attack plugins corresponding to capability-triggered attack samples via the plugin manager means that the plugin manager retrieves and loads the corresponding attack plugins from the plugin repository based on the function type identifiers of each capability-triggered attack sample in the test task sequence. For example, if the test task sequence contains 6 capability-triggered attack samples, where 2 have the function type identifier of code generation (the plugin manager loads a code injection detection plugin), 2 have the function type identifier of tool invocation (the plugin manager loads a tool unauthorized invocation plugin), and 2 have the function type identifier of multi-turn dialogue (the plugin manager loads a context inducement plugin). Each attack plugin is deployed in the plugin repository as a dynamic link library. The plugin manager parses the interface adaptation information in the plugin declaration file, verifies that the plugin's input and output specifications are consistent with the data format of the capability-triggered attack samples, and then completes the loading.

[0035] The fact that attack plugins run in independent process spaces means that each attack plugin is assigned to an independent process space. This independent process space is an isolated runtime space built based on container technology, which only allows three types of system call permissions: network communication, read-only file access, and standard output. Write operations to the host system directory are prohibited, the creation of child processes is prohibited, and access to network endpoints other than the target large language model call address is prohibited. For example, container instance A is allocated to the code injection detection plugin, container instance B is allocated to the tool privilege escalation call plugin, and container instance C is allocated to the multi-turn dialogue inducement plugin. The three container instances are isolated from each other, and the resource limits for each are set to 1 CPU core, 512 megabytes of memory, and 1 gigabyte of storage.

[0036] The plugin manager periodically receives runtime status verification values ​​reported by independent process spaces. This means that the plugin manager sends status query requests to each independent process space at preset intervals, and each independent process space collects and reports its current runtime status data upon receiving the request. Runtime status verification values ​​include memory usage, CPU utilization, abnormal exit count, and input data length. For example, if the status query period is set to 5 seconds, container instance A, on the Nth query, reports a memory usage of 480 megabytes, a CPU utilization of 35%, an abnormal exit count of 0, and a response content length of 2048 bytes.

[0037] Specifically, when an execution anomaly is detected during the processing of the response content returned by the target large language model by the attack plugin, freezing the independent process space and marking the corresponding capability-triggered attack sample as a high-risk sample means that the plugin manager compares the reported runtime status verification value with preset anomaly thresholds. When any indicator exceeds the threshold, an execution anomaly is determined to have occurred. For example, a memory usage threshold of 512 megabytes, a CPU utilization threshold of 80%, and an abnormal exit count threshold of 1 are set. If container instance A's memory usage rises to 530 megabytes while processing a certain response content, exceeding the memory usage threshold, the plugin manager immediately sends a freeze command to container instance A, stopping all processes within that container instance. At the same time, the currently processed capability-triggered attack sample is marked as a high-risk sample, and its sample identifier is written to the high-risk mark list.

[0038] The storage of high-risk samples in the isolated partition of the attack sample library refers to the plugin manager removing attack data marked as high-risk samples from the test task sequence and transferring it to a pre-defined isolated partition within the attack sample library. This isolated partition is physically isolated from the normal sample partition, and its storage path is the isolated subdirectory under the root directory of the attack sample library. Read access to this subdirectory is only granted to the security audit module, and write access is only granted to the plugin manager. For example, the aforementioned marked high-risk samples were originally stored in the code-generation partition of the attack sample library. After being marked, the plugin manager migrates them to the isolated / code-generation subdirectory, retaining the migration record in the original location to prevent subsequent test tasks from repeatedly loading the sample.

[0039] This application's solution achieves secure isolation between the test execution process and the host system by allocating independent process spaces with restricted system call permissions to each attack plugin. Even if the response returned by the target large language model contains maliciously crafted countermeasure data, the damage can be limited to a single container instance. Simultaneously, by periodically collecting runtime status verification values ​​and immediately freezing the execution environment in case of anomalies, the risk of resource exhaustion or code execution triggered by attack plugins handling abnormal responses can be promptly prevented. It is precisely because the independent process space provides dual boundaries of resources and permissions, and the runtime status verification mechanism provides real-time anomaly detection capabilities, that the testing system can quickly isolate the threat source when encountering high-risk attack samples, preventing high-risk samples from contaminating the normal attack sample library or damaging the test host environment, thereby ensuring the continuous and stable operation of automated red team attack testing.

[0040] S105, send each capability-triggered attack sample in the test task sequence to at least two target large language models in parallel, and receive the response content returned by each target large language model.

[0041] Furthermore, in S105, receiving the response content returned by each target large language model includes: performing structured parsing on the response content returned by each target large language model, and extracting the function call sequence information in each response content. The function call sequence information includes the external interface identifiers and call order called by the target large language model during the processing of the attack sample.

[0042] Specifically, sending the capability-triggered attack samples from the test task sequence in parallel to at least two target large language models means that the test system simultaneously distributes the test task sequence assembled in step S103 to multiple target model nodes through a concurrent scheduling mechanism. For example, the first target large language model is deployed on service node A, with its calling address being https: / / demo-test-node.virtual-lab.local / v1 / inference; the second target large language model is deployed on service node B, with its calling address being https: / / demo-test-node.virtual-lab.local / v2 / inference. The test system initializes the number of concurrent channels to 2 and simultaneously sends the test task sequences T1 to T6 to service node A and service node B. During the sending process, service node A and service node B independently receive and process each capability-triggered attack sample, generating corresponding response content respectively.

[0043] The structured parsing of the response content returned by each target language model refers to the test system extracting field information with a fixed format from the raw response data returned by each target model according to preset response message parsing rules. For example, the response content RA returned by the first target language model is a JSON format message containing status, content, and execution-trace fields; the response content RB returned by the second target language model is also in JSON format, containing state, output, and tool-log fields. The test system maps fields to RA and RB respectively, using the execution-trace and tool-log fields as the source fields for function call information.

[0044] Extracting function call sequence information from each response content refers to extracting record data of the target large language model's calls to external services or tools during the processing of attack samples from the parsed source fields. Function call sequence information includes external interface identifiers and call order. For example, parsing the `execution-trace` field of the first target large language model yields two call records: the first record's external interface identifier is `code-executor-v1`, and its call order is 1; the second record's external interface identifier is `file-access-v1`, and its call order is 2. Parsing the `tool-log` field of the second target large language model yields one call record, whose external interface identifier is `code-executor-v1`, and its call order is 1. Thus, the function call sequence information for the first target large language model is `code-executor-v1` first, followed by `file-access-v1`, while the function call sequence information for the second target large language model only contains `code-executor-v1`.

[0045] The external interface identifier refers to the unique name code of the external tool or service invoked by the target large language model when processing a request. It is usually assigned by the interface provider during system registration. For example, code-executor-v1 represents the first version of the code execution service interface, file-access-v1 represents the first version of the file access service interface, and search-engine-v2 represents the second version of the search service interface. The invocation order refers to the order in which the external interfaces are invoked during the same response generation process, represented by a positive integer; the smaller the value, the earlier the invocation. For example, invocation order 1 indicates that this interface is the first external interface invoked during this request processing, and invocation order 2 indicates that this interface is the second external interface invoked.

[0046] When the response content is unstructured natural language text, the structured parsing includes: performing regular expression-based pattern matching on the response text, with matching rules including a predefined interface call feature lexicon and call order sentence templates; for cases where matching fails, calling a pre-trained lightweight sequence labeling model to identify external interface identifiers and call order in the text, with the sequence labeling model outputting interface entities and their order in a BIO labeling manner; if the above methods fail to extract valid function call sequence information, the response is marked as a no-function-call response, and a deviation baseline under the no-call condition is added to the expected response features of the corresponding capability-triggered attack sample.

[0047] This embodiment achieves synchronous testing among multiple models by sending test task sequences in parallel to at least two target large language models. Simultaneously, by performing structured parsing on each response and extracting function call sequence information, external dependency behavior data of the target models when processing attack samples is obtained. It is precisely because the parallel sending mechanism ensures the synchronous execution of multi-model tests, and the structured parsing mechanism reveals the internal function call paths of the models, that subsequent steps can accurately compare the differences in function calls of different models for the same test task sequence, thereby identifying defensive weaknesses in a specific model architecture.

[0048] S106, extract function call information from each response content, update the model capability identifier set according to the function call information, and adjust the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set; Further, in S106, the step of extracting function call information from each response content and updating the model capability identifier set according to the function call information includes: The function call exception type is parsed from the function call sequence information, and a deep probing rule is selected from the policy library according to the function call exception type. The capability trigger attack sample corresponding to the deep probing rule is added to the test task sequence. Execute the additional test task sequence, obtain the supplementary response content returned by each target large language model, extract supplementary function call information from the supplementary response content, and update the model capability identifier set according to the supplementary function call information.

[0049] Specifically, function call information is extracted from each response. This involves the test system performing anomaly analysis on the function call sequence information parsed in step S105, extracting call records that exceed the normal behavior baseline. For example, if the normal behavior baseline is set to the attack sample corresponding to the function type identifier `code-generation`, it is expected that only the external interface identifier `code-executor-v1` will be triggered, and the call count will be one, with the call parameters limited to the range of user-input code snippets. In response R-A1, the first target large language model triggers two external interfaces, `code-executor-v1` and `file-access-v1`, in call order 1 and 2 respectively, and the input parameter for `file-access-v1` includes a system-level directory path. In response R-B1, the second target large language model only triggers `code-executor-v1`, in call order 1, with normal parameter range. Therefore, unexpected file access anomalies are identified from the function call sequence information of the first target large language model.

[0050] The process involves parsing function call exception types from the function call sequence information. This involves comparing the extracted function call information with a preset baseline of normal behavior. When the actual call record deviates from the baseline, it is classified into a specific exception type. Function call exception types include unexpected interface calls, abnormal parameter passing, unauthorized call order, and out-of-scope repeated calls. For example, the aforementioned first target large language model called the file-access-v1 interface, which was not declared in the initial model capability identifier set, and this call is not directly related to the code-generation function type, thus it is parsed as an unexpected interface call exception type. At the same time, the file-access-v1 input parameter contains a system-level directory path, which exceeds the file access range required for normal code execution, further parsing it as an abnormal parameter passing exception type.

[0051] The system selects deep probing rules from the policy library based on the anomaly type of the function call. Specifically, the testing system retrieves the corresponding deep probing rule entry from the policy library based on the parsed anomaly type code. The policy library stores deep probing rules in partitions according to anomaly type codes. Each rule entry includes a rule identifier, triggering conditions, and a corresponding attack sample template identifier. For example, for the unexpected interface call anomaly type, the policy library stores an interface privilege escalation detection rule, identified as R001. This rule indicates the addition of an attack sample that can induce the target model to perform sensitive operations through exposed unexpected interfaces. For the abnormal parameter transmission anomaly type, the policy library stores a parameter boundary detection rule, identified as R002. This rule indicates the addition of an attack sample containing malformed or excessively long parameters. For the first target large language model, two deep probing rules, R001 and R002, are selected from the policy library based on its unexpected interface call anomaly type and abnormal parameter transmission anomaly type.

[0052] This process involves appending attack samples corresponding to the deep probing rules to the test task sequence. Specifically, based on the attack sample template identifier associated with the selected deep probing rule, specific attack sample data is retrieved from the attack sample library and inserted at the end of the current test task sequence. For example, the attack sample template identifier associated with rule R001 is S-001, and the corresponding sample is inducement text requesting the model to read the system log directory through the code execution interface; the attack sample template identifier associated with rule R002 is S-002, and the corresponding sample is malformed parameter text injecting nested system commands into the code execution interface. S-001 and S-002 are then appended as supplementary attack samples T7 and T8 to the end of the test task sequence of the first target large language model, forming the updated test task sequence T1 to T8.

[0053] The test system executes the appended test task sequence and obtains supplementary response content from each target large language model. Specifically, the test system sends the appended test task sequence to the corresponding target model and waits to receive the supplementary responses. For example, T7 and T8 are sent sequentially to the first target large language model, obtaining supplementary responses R-A7 and R-A8. For the second target large language model, since no function call anomaly is triggered, no supplementary attack samples are added; only the original test task sequence execution results are maintained.

[0054] This involves extracting supplementary function call information from the supplementary response content. Specifically, the supplementary response undergoes the same structured parsing as step S105 to extract the supplementary function call sequence information. For example, parsing supplementary response R-A7 yields external interface identifiers file-access-v1 and network-access-v1, with call orders of 1 and 2 respectively. The access path of file-access-v1 points to the system network configuration file. Parsing supplementary response R-A8 yields external interface identifier code-executor-v1, with a call order of 1, and its execution parameters include a system Shell command call marker.

[0055] Specifically, the model capability identifier set is updated based on supplementary function call information. This involves adding new function types or abnormal functional behaviors revealed in the supplementary function call information to the original model capability identifier set as function type identifiers and marking them as risky. For example, the initial model capability identifier set includes code-generation and tool-invocation. Based on the supplementary function call information in R-A7, file-access and network-access are added as new function type identifiers to the model capability identifier set of the first target large language model and marked as abnormally exposed. Based on the supplementary function call information in R-A8, the code-generation function type identifier is confirmed to have parameter injection risk, and a parameter injection risk marker is added to its metadata. The updated model capability identifier set is used to adjust the selection strategy for subsequent capability-triggered attack samples, ensuring that the sample selection module prioritizes attack samples related to file-access and network-access in subsequent tests and strengthens parameter boundary testing for code-generation type samples.

[0056] This embodiment achieves dynamic adaptation of the testing strategy to the actual response behavior of the model by parsing the exception type from the function call sequence information and triggering the automatic selection of deep probing rules. Simultaneously, by executing additional test task sequences and extracting supplementary function call information, the understanding of the target model's capability boundaries is continuously expanded. It is precisely because the deep probing rules can perform targeted and in-depth testing on discovered abnormal call behaviors, and because the supplementary function call information can continuously correct the model's capability identifier set, that the testing system can quickly converge to the true functional boundaries and security vulnerabilities of the target large language model within a limited number of rounds, avoiding the coverage blind spots caused by blind testing based on static capability declarations.

[0057] S107, compare each response content with the expected response characteristics to determine the security test results of each target large language model under the function type identifier.

[0058] Furthermore, in S107, the comparison of each response content with the expected response characteristics is performed, such as... Figure 2 As shown, it includes: S2001, extract the text semantic deviation, response temporal fluctuation, and response structural integrity of each response content; Specifically, the semantic deviation of each response is extracted. This involves comparing the semantic vectors of the response text returned by the target large language model with the security rejection template in the expected response features, and calculating the degree of deviation. For example, the security rejection template for code generation attack samples in the expected response features includes baseline semantic vectors such as "refuse to execute," "cannot provide," and "does not comply with security policies." The response text returned by the first target large language model for attack sample P2 is "can provide a sample code." This response text is processed by word segmentation and vectorization to obtain the actual semantic vector. The cosine distance between the actual semantic vector and the semantic vector of the security rejection template is calculated, resulting in a text semantic deviation of 0.85. The closer this value is to 1, the greater the degree of deviation from the security rejection pattern.

[0059] This involves extracting response time volatility, which compares the response time of the target large language model processing the attack sample with the baseline value of the normal request processing time, and calculates the relative volatility. For example, if the baseline normal request processing time T0 is set to 500 milliseconds, and the response time T1 of the first target large language model processing the attack sample P2 is 1200 milliseconds, then the response time volatility V... time Calculate using the following formula: V time = |T1 - T0| / T0 In the formula, V time This represents the response time fluctuation, where T1 represents the actual response time and T0 represents the baseline normal request processing time. Substituting the values, we get V. time = |1200 - 500| / 500 = 1.4. This value is greater than 1, indicating that the actual response time significantly exceeds the normal baseline, and there is a timing anomaly caused by processing unusually complex requests or triggering additional background calculations.

[0060] The process of extracting response structure integrity involves matching the fields of the response message returned by the target large language model with the expected response structure template and calculating the structure integrity ratio. For example, the expected response structure template requires the response message to contain three necessary fields: a status field, a content field, and an execution trace field. The response message returned by the first target large language model contains a status field and a content field, but lacks an execution trace field. The actual number of fields is 2, while the expected number of fields is 3, resulting in a response structure integrity ratio C. struct Calculate using the following formula: C struct = Nactual / N expected In the formula, C struct Indicates the structural integrity of the response, N actual N represents the number of necessary fields actually included in the response message. expected This represents the number of required fields in the expected response structure template. Substituting the values, we get C. struct = 2 / 3 ≈ 0.67. The closer this value is to 0, the more severe the missing response structure, which may mean that the target model interrupted the standard response process during anomaly handling.

[0061] S2002, the text semantic deviation, the response temporal volatility, and the response structural integrity are weighted and fused to obtain a comprehensive risk value; Specifically, a weighted fusion calculation is performed on the text semantic deviation, response temporal fluctuation, and response structural integrity. That is, weight coefficients are assigned according to the sensitivity of each dimension to the security risk representation, and the values ​​of each dimension are linearly weighted to obtain the comprehensive risk value R. The comprehensive risk value R is calculated according to the following formula: R = w1 × D text + w2 × V time + w3 × C struct In the formula, R represents the comprehensive risk value, w1 represents the weighting coefficient of the text semantic deviation, and D... text V represents the semantic deviation of the text, w2 represents the weighting coefficient of the response time-series volatility, and V time w3 represents the response time series volatility, w3 represents the weighting coefficient for response structural integrity, and C represents the response time series volatility. struct This indicates the integrity of the response structure. In practical applications, w1 is set to 0.5, w2 to 0.3, and w3 to 0.2. The first target large language model targets the D of attack sample P2. text It is 0.85, V time It is 1.4, C struct The value is 0.67. Substituting this into the formula, we get: R = 0.5 × 0.85 + 0.3 × 1.4 + 0.2 × 0.67 = 0.425 + 0.420 + 0.134 =0.979.

[0062] S2003, when the comprehensive risk value exceeds the preset risk threshold, it is determined that there is a security risk under the function type identifier.

[0063] Specifically, a security risk is determined when the overall risk value exceeds a preset risk threshold. This involves comparing the overall risk value calculated in step S2002 with a pre-set risk assessment threshold. For example, the preset risk threshold R-th is set to 0.75. The overall risk value R of the first target large language model against attack sample P2 is 0.979. Since 0.979 is greater than 0.75, it is determined that this target model has a security risk under the code generation function type identifier. The second target large language model against the same attack sample P2 has a text semantic deviation of 0.15, a response temporal fluctuation of 0.1, and a response structural integrity of 1.0. The calculated overall risk value is 0.5×0.15+0.3×0.1+0.2×1.0=0.075+0.03+0.2=0.305. This value is less than 0.75, therefore it is determined that the second target model did not trigger a security risk under the same function type identifier.

[0064] Furthermore, in S107, determining the security test results of each target large language model under the functional type identifier includes: The semantic similarity of the responses of each target large language model to the same related attack chain is compared to identify abnormal response patterns that only appear in specific target large language models. Obtain the model architecture type identifier corresponding to each target large language model, and perform attribution analysis on the abnormal response pattern based on the model architecture type identifier to determine the source of potential vulnerabilities.

[0065] Specifically, the semantic similarity of the responses from each target large language model to the same associated attack chain is compared. This involves calculating the similarity in vector space between the response texts returned by the first and second target large language models for the same associated attack chain P1 pointing to P2. For example, the response R-A2 returned by the first target large language model contains executable code snippets and system interface call descriptions, while the response R-B2 returned by the second target large language model is a standard denial-of-service warning. After semantically vectorizing R-A2 and R-B2, a cosine similarity score of 0.25 is obtained, which is lower than the preset similarity threshold of 0.6, indicating a significant difference between the two models' responses to the same attack chain. Further comparison of R-A2 with the expected security denial template confirms that R-A2 contains an anomalous response pattern that only exists in the response of the first target large language model, namely, executable code output containing system interface call descriptions.

[0066] Attribution analysis was performed on the abnormal response patterns by obtaining the model architecture type identifiers corresponding to each target large language model. This involved mapping the identified abnormal response patterns to the architectural characteristics of the target models to infer the technical root cause of the vulnerabilities. For example, the first target large language model's model architecture type identifier is MoE architecture, which uses an expert routing mechanism to distribute input requests to different expert subnetworks for processing. The second target large language model's model architecture type identifier is Dense Transformer architecture, which uses a fully connected feedforward network to uniformly process all inputs. Attribution analysis results show that the abnormal response patterns in the first target large language model are related to the abnormal activation of a specific expert subnetwork responsible for code-type task processing in the MoE architecture. This expert subnetwork may have been exposed to pre-training data containing system-level code execution during the training phase, causing it to break the security alignment strategy constraints when facing deceptive attack samples, resulting in executable code output containing system interface calls. Therefore, the potential source of the vulnerability was determined to be pre-training data contamination. The second target large language model, due to its Dense Transformer architecture and lack of a separate code-type expert subnetwork, maintained a secure denial response under the same attack chain and did not trigger abnormal response patterns.

[0067] This embodiment quantifies multi-dimensional response features into a single comprehensive risk value by weighted and fused calculation of text semantic deviation, response temporal fluctuation, and response structural integrity, thus achieving numerical determination of security risks. Simultaneously, by comparing the semantic similarity of responses from multiple models and combining this with attribution analysis based on model architecture type identification, the test results are extended from risk discovery to vulnerability root cause localization. Because the comprehensive risk value can uniformly represent the degree of response anomalies across different dimensions, and because attribution analysis can correlate and map abnormal response patterns with model architecture characteristics, the testing system can not only identify whether a target large language model has security risks, but also further determine whether the risk is related to a specific model architecture implementation, providing a directional basis for subsequent model optimization and security hardening.

[0068] Furthermore, the method also includes: generating a risk transmission path diagram between models based on the security test results of each target large language model under the functional type identifier. The risk transmission path diagram between models includes the risk output association path between each target large language model and the transmission strength of the risk on the association path. The risk output association path is determined based on the relationship between the response content of each target large language model to the same capability-triggered attack sample.

[0069] Specifically, generating a risk propagation path diagram between models refers to the process by which, after the testing system determines the security test results for each target model in step S107, it extracts risk-related entities from the responses of each model to the same attack sample. By analyzing the co-occurrence relationships of entities between different model responses, it constructs a risk propagation topology between models. For example, three target large language models are defined as the first target model NodeA, the second target model NodeB, and the third target model NodeC. In step S105, all three models perform parallel attack chain tests on the same capability-triggered attack sample P2, and the security test results are determined separately in step S107.

[0070] Extracting risk-related entities from the response content of each target large language model to the same capability-triggered attack sample refers to performing structured entity parsing on the response content that has been identified as having security risks, and extracting key information units directly related to the risky behavior. For example, the response R-A2 of the first target model NodeA has been identified as having a security risk under the code generation function type identifier. The risk-related entities extracted from R-A2 include the external interface identifier file-access-v1, the system directory path / var / log / system.log, and the executable code flag exec-flag, forming a risk entity set EA, with a total of 3 elements Nsource. The response R-B2 of the second target model NodeB is determined to be safe, and its response content is a standard rejection text. No risk-related entities were extracted, and the risk entity set EB is empty. The response R-C2 of the third target model NodeC is determined to have a security risk. The risk-related entities extracted from R-C2 include the external interface identifier file-access-v1, the system directory path / var / log / system.log, and the retrieval reference flag ref-flag, forming a risk entity set EC, with a total of 3 elements.

[0071] The determination of risk output association paths based on the correlation between the response content of each target large language model to the same capability-triggered attack sample involves performing an intersection operation on the risk entity sets of each model. When the response of the target model contains a risk entity from the source model, a risk output association path is established from the source model to the target model. For example, performing an intersection operation on the risk entity set EA of the first target model NodeA and the risk entity set EC of the third target model NodeC yields the common entities file-access-v1 and / var / log / system.log, with a total of 2 intersection elements, Ncommon. Since the response of the third target model NodeC contains a risk entity from the first target model NodeA, a risk output association path is established from NodeA to NodeC, denoted as Path-AC. The risk entity set of the second target model NodeB is empty, and its response does not contain any risk entities from other models; therefore, no risk output association path is established with NodeB as the starting or ending point.

[0072] The transmission strength of risk along the associated path refers to the degree to which the risk output of the source model is inherited or referenced by the response of the target model in the risk output associated path. The transmission strength S is calculated using the following formula: S = k × Ncommon / Nsource In the formula, S represents the transmission strength, k represents the semantic association correction coefficient, used to correct the semantic association bias that may be caused by simple entity counting, and its value ranges from 0 to 1; Ncommon represents the number of risk entities in the target model response that are the same as the risk entities in the source model; Nsource represents the total number of entities in the risk entity set of the source model. In practical applications, the semantic association correction coefficient k is set to 0.9, the total number of risk entities in the risk entity set of the first target model NodeA is 3, and the number of risk entities shared by the third target model NodeC and NodeA is 2. Substituting these values ​​into the formula, we get: S = 0.9 × 2 / 3 = 0.6 The transmission strength value of 0.6 indicates that 60% of the risk entities in the risk output of the first target model NodeA are inherited by the response of the third target model NodeC, indicating a moderately strong risk transmission relationship between the two. When the transmission strength S is greater than the preset transmission threshold of 0.5, the test system marks this path as a high-risk transmission path, represented by a bold directed line segment in the inter-model risk transmission path diagram.

[0073] The risk propagation path graph between models refers to a directed graph structure with each target large language model as a node, the risk output association path as directed edges, and the propagation strength as the edge weight. For example, the risk propagation path graph between the three target models mentioned above contains three nodes NodeA, NodeB, and NodeC, and a directed edge Path-AC from NodeA to NodeC, with an edge weight of 0.6. NodeB is an isolated node, indicating that this model did not participate in risk propagation under the current attack sample.

[0074] This application's solution extracts risk entities from the response content of each model and analyzes their co-occurrence relationships, thus linking the originally independent multi-model security test results into a risk propagation topology. Simultaneously, it quantifies the degree of risk inheritance between models by calculating the propagation strength. Because the risk output association path can reveal the risk content inheritance relationship between different target large language models under the same attack context, and because the propagation strength can quantify the strength of this inheritance relationship, the testing system can not only identify security vulnerabilities in a single model but also discover the risk cascading effect in multi-model deployment scenarios, providing a directional basis for constructing risk isolation strategies between models.

[0075] Secondly, this embodiment also proposes an automated testing system for red team attacks targeting large language models, such as... Figure 3 As shown, it includes: Configuration module 301 obtains the interface configuration information of the target large language model, the interface configuration information including the calling address and authentication parameters; The identification module 302 obtains a set of model capability identifiers supported by the target large language model, wherein the set of model capability identifiers includes functional type identifiers possessed by the target large language model; Module 303 selects capability-triggered attack samples corresponding to the function type identifier from the attack sample library according to the model capability identifier set, and assembles the selected capability-triggered attack samples into a test task sequence. Loading module 304 is used to load attack plugins corresponding to the capability-triggered attack samples through the plugin manager; The test module 305 sends the capability-triggered attack samples in the test task sequence to at least two target large language models in parallel, and receives the response content returned by each target large language model. The update module 306 extracts function call information from each response content, updates the model capability identifier set according to the function call information, and adjusts the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set. The comparison module 307 compares each response content with the expected response characteristics to determine the security test results of each target large language model under each threat type.

[0076] This system can be used to execute the automated testing method for red team attacks targeting large language models described in the first aspect, which will not be elaborated further here.

[0077] Thirdly, this embodiment also proposes an electronic device comprising: one or more processors, and a memory for storing one or more computer programs; the computer programs are configured to be executed by the one or more processors, and the programs include steps for performing an automated testing method for red team attacks targeting large language models as described in the first aspect, which will not be repeated here.

[0078] In addition, the electronic device includes a communication interface for data interaction with external devices. The processor may be at least one of a central processing unit, a graphics processing unit, or a dedicated neural network acceleration chip, for providing computing power support; the memory includes non-volatile storage media and random access memory, for persistently storing computer programs and temporarily caching test task sequences, model capability identifier sets, and response content.

[0079] The electronic device also includes a display unit for displaying the interface configuration information of the target large language model, the set of model capability identifiers, the assembly status of the associated attack chain, the summary of the response content returned by each target large language model, the extracted function call sequence information, the calculated comprehensive risk value, the security test result judgment conclusion, and the risk transmission path diagram between models.

[0080] Fourthly, this embodiment also proposes a storage medium storing a computer program; characterized in that the program is loaded and executed by a processor to implement the steps of the automated testing method for red team attacks targeting large language models as described in the first aspect. Further details will not be elaborated here.

[0081] This embodiment provides a non-transient computer-readable storage medium storing a computer program. The storage medium may be at least one of a read-only memory, flash memory, hard disk, optical disk, or removable storage device, used for persistently storing the computer program and its related data.

[0082] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for red team attack automation testing for large language models, characterized in that, include: Obtain the interface configuration information of the target large language model, wherein the interface configuration information includes the calling address and authentication parameters; Obtain the set of model capability identifiers supported by the target large language model, wherein the set of model capability identifiers includes the functional type identifiers possessed by the target large language model; Based on the set of model capability identifiers, select capability-triggered attack samples corresponding to the function type identifiers from the attack sample library, and assemble the selected capability-triggered attack samples into a test task sequence. Load the attack plugin corresponding to the attack sample triggered by the aforementioned capability through the plugin manager; Send each capability-triggered attack sample in the test task sequence to at least two target large language models in parallel, and receive the response content returned by each target large language model; Extract function call information from each response content, update the model capability identifier set according to the function call information, and adjust the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set; By comparing each response content with the expected response characteristics, the security test results of each target large language model under the stated function type identifier are determined.

2. The automated testing method for red team attacks targeting large language models according to claim 1, characterized in that, The step of assembling the selected capability-triggered attack samples into a test task sequence includes: Identify context preservation markers in the response content of preceding capability-triggered attack samples, wherein the context preservation markers include session state identifiers or history reference indexes; Based on the context-preserving markers, construct subsequent capability-triggered attack samples and assemble the preceding and subsequent capability-triggered attack samples into an associated attack chain.

3. The automated testing method for red team attacks targeting large language models according to claim 1, characterized in that, The process of loading the attack plugin corresponding to the attack sample triggered by the capability through the plugin manager includes: The attack plugin runs in an independent process space, which has limited system call permissions; The plugin manager periodically receives the running status verification value reported by the independent process space. When an execution exception is detected during the processing of the response content returned by the target large language model by the attack plugin, the independent process space is frozen and the corresponding capability trigger attack sample is marked as a high-risk sample. The high-risk sample is then stored in the isolation partition of the attack sample library.

4. The automated testing method for red team attacks targeting large language models according to claim 2, characterized in that, After assembling the selected capability-triggered attack samples into a test task sequence, the associated attack chain is sent in parallel to at least two target large language models, so that each target large language model responds based on the attack history context of the associated attack chain.

5. The automated testing method for red team attacks targeting large language models according to claim 4, characterized in that, The process of receiving the response content returned by each target large language model includes: The response content returned by each target large language model is subjected to structured parsing to extract the function call sequence information in each response content. The function call sequence information includes the external interface identifiers and call order called by the target large language model in the process of processing the attack sample.

6. The automated testing method for red team attacks targeting large language models according to claim 5, characterized in that, The step of extracting function call information from each response content and updating the model capability identifier set based on the function call information includes: The function call exception type is parsed from the function call sequence information, and a deep probing rule is selected from the policy library according to the function call exception type. The capability trigger attack sample corresponding to the deep probing rule is added to the test task sequence. Execute the additional test task sequence, obtain the supplementary response content returned by each target large language model, extract supplementary function call information from the supplementary response content, and update the model capability identifier set according to the supplementary function call information.

7. The automated testing method for red team attacks targeting large language models according to claim 6, characterized in that, The comparison of each response content with the expected response characteristics includes: Extract the text semantic deviation, response temporal volatility, and response structural integrity from each response content; the response temporal volatility is calculated based on the timestamp field in the response content and the request sending time. A comprehensive risk value is obtained by weighted and fused calculation of the text semantic deviation, the response temporal volatility, and the response structural integrity. When the overall risk value exceeds the preset risk threshold, it is determined that there is a security risk under the function type identifier.

8. The automated testing method for red team attacks targeting large language models according to claim 7, characterized in that, The determination of the security test results of each target large language model under the functional type identifier includes: The semantic similarity of the responses of each target large language model to the same related attack chain is compared to identify abnormal response patterns that only appear in specific target large language models. Obtain the model architecture type identifier corresponding to each target large language model, and perform attribution analysis on the abnormal response pattern based on the model architecture type identifier to determine the source of potential vulnerabilities.

9. The automated testing method for red team attacks targeting large language models according to claim 8, characterized in that, The method further includes: generating a risk transmission path diagram between models based on the security test results of each target large language model under the functional type identifier. The risk transmission path diagram between models includes the risk output association path between each target large language model and the transmission strength of the risk on the association path. The risk output association path is determined based on the correlation of the response content of each target large language model to the same capability-triggered attack sample.

10. An automated testing system for red team attacks targeting large language models, characterized in that, include: The configuration module obtains the interface configuration information of the target large language model, including the calling address and authentication parameters. The identification module obtains a set of model capability identifiers supported by the target large language model, wherein the set of model capability identifiers includes functional type identifiers possessed by the target large language model; The selection module selects capability-triggered attack samples corresponding to the function type identifier from the attack sample library based on the model capability identifier set, and assembles the selected capability-triggered attack samples into a test task sequence. A loading module is used to load attack plugins corresponding to the attack samples triggered by the aforementioned capability through a plugin manager. The testing module sends attack samples triggered by each capability in the test task sequence to at least two target large language models in parallel, and receives the response content returned by each target large language model. The update module extracts function call information from each response content, updates the model capability identifier set according to the function call information, and adjusts the selection strategy of subsequent capability-triggered attack samples based on the updated model capability identifier set. The comparison module compares each response content with the expected response features to determine the security test results of each target large language model under the stated function type identifier.

Images