Big model red team test data processing method and system fusing prior safety rules

Abstract

The present application belongs to the technical field of large model security test data processing, and relates to a large model red team test data processing method and system fusing prior security rules, the system comprising a rule boundary generation module, a link association processing module, a boundary state determination module, a red team test generation module, a test execution collection module, a boundary crossing identification attribution module and a test result processing module; the method comprising: acquiring security boundary configuration data containing prior security rules and generating rule boundary data; collecting interactive link data and generating interactive link association data; determining the knowledge boundary state corresponding to the test subject and generating multi-stage red team test tasks; collecting test execution link data after executing the test, identifying boundary crossing content that exceeds the use boundary, determining the corresponding knowledge source and trigger path, and generating red team test result data to represent the knowledge content boundary crossing output risk of the large model object being tested.

Description

Technical Field

[0001] This invention relates to the field of large-scale model security test data processing technology, and in particular to a method and system for processing large-scale model red team test data that integrates prior security rules. Background Technology

[0002] With the application of large-scale model technology in scenarios such as enterprise knowledge question answering, document retrieval enhanced question answering, business intelligence agents, and plug-in invocation-based question answering, the large-scale model object under test typically needs to call external knowledge sources such as enterprise knowledge bases, document libraries, databases, knowledge graphs, or business system interfaces, and generate model responses in combination with the interaction context. This type of application can improve the relevance between model answers and business data, but it also makes the model output subject to the combined influence of factors such as the test subject's permissions, knowledge object usage restrictions, interaction context state, and model response output boundaries.

[0003] In practical applications, a test subject may be allowed access to certain knowledge objects, but not to restricted knowledge objects related to its business. For example, a test subject may access project progress summaries, general contract terms, or customer communication records, but should not obtain base prices, profit margin fields, internal assessment conclusions, or undisclosed business strategies. However, during multiple rounds of interaction, the large model object under test may first invoke accessible knowledge objects, then generate a response by combining context or related knowledge objects, and finally output content corresponding to restricted knowledge objects through summarization, rewriting, induction, or inference. In this case, although the model response may not directly reproduce the original text of the restricted knowledge object, it may still pose a risk of knowledge content being output out of bounds.

[0004] Existing red team testing methods for large models typically rely on manually constructed attack warning words, sensitive question libraries, or violation output detection rules, focusing more on whether the final response of the large model being tested contains obviously sensitive content. This approach struggles to uniformly correlate the usage boundaries between prior security rules, external knowledge sources, knowledge objects, interaction contexts, and model responses, and also finds it difficult to automatically generate progressively related red team test tasks based on the knowledge boundary state corresponding to the test subject.

[0005] Therefore, when the out-of-bounds content of the large model object under test is triggered by the knowledge call process, context changes and multi-stage test inputs, the existing testing methods can often only determine that there is a suspected risk in the model response, and it is difficult to further determine the knowledge source and triggering path corresponding to the out-of-bounds content. This results in the red team test results lacking traceability and making it difficult to provide an effective data basis for subsequent out-of-bounds pattern analysis or test task updates. Summary of the Invention

[0006] The primary objective of this invention is to provide a large-scale red team test data processing system that integrates prior safety rules. This system has the advantages of automatically generating multi-stage red team test tasks and determining the corresponding knowledge source and trigger path when identifying knowledge content that exceeds the boundary, thereby improving the traceability and relevance of large-scale red team test results.

[0007] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A large-scale red team test data processing system integrating prior safety rules is used to perform red team test data processing on the large-scale model object under test, wherein the large-scale model object under test is configured to call external knowledge sources, and the system includes: The rule boundary generation module is used to obtain security boundary configuration data containing prior security rules, and generate rule boundary data based on the security boundary configuration data. The rule boundary data is used to characterize the usage boundary between the test subject, knowledge object, interaction context and the model response of the tested large model object. The link association processing module is used to collect the interaction link data during the interaction process of the large model object under test, and to associate and organize the interaction link data according to the rule boundary data to generate interaction link association data. The boundary state determination module is used to determine the knowledge boundary state corresponding to the test subject based on the rule boundary data and the interaction link association data, and generate boundary state data. The red team test generation module is used to generate multi-stage red team test tasks based on the boundary state data, and to generate red team test task data based on the multi-stage red team test tasks. The test execution acquisition module is used to input the red team test task data into the large model object under test, and to collect test execution link data; The boundary violation identification and attribution module is used to identify, based on the rule boundary data, the boundary state data, and the test execution link data, out-of-boundary content output by the tested large model object that exceeds the usage boundary represented by the rule boundary data, and to determine the knowledge source and triggering path corresponding to the out-of-boundary content, and generate out-of-boundary attribution data. The test result processing module is used to generate red team test result data based on the out-of-bounds attribution data. The red team test result data is used to characterize the risk of out-of-bounds output of knowledge content of the tested large model object.

[0008] Furthermore, the rule boundary generation module includes a boundary configuration parsing unit and a boundary constraint construction unit; The security boundary configuration data includes at least one of the following: prior security rules, test subject constraint information, knowledge object permission information, interaction context constraint information, and model response constraint information; The prior security rules include at least one of the following: external knowledge source access permission rules, knowledge object usage restriction rules, context call restriction rules, and model response output restriction rules; The boundary configuration parsing unit is used to extract test subject permission data, knowledge object boundary attribute data, interaction context constraint data, and model response constraint data from the security boundary configuration data. The boundary constraint construction unit is used to generate multiple boundary constraint items based on the test subject permission data, the knowledge object boundary attribute data, the interaction context constraint data, and the model response constraint data, wherein the rule boundary data includes multiple boundary constraint items. Each of the boundary constraint items includes a subject constraint field, a knowledge object constraint field, a context constraint field, and a response constraint field; The subject constraint field is used to characterize the permission identity of the corresponding test subject, the knowledge object constraint field is used to characterize the usage restrictions of the corresponding knowledge object, the context constraint field is used to characterize the knowledge call restrictions under the corresponding interaction context, and the response constraint field is used to characterize the output restrictions when the tested large model object generates a model response.

[0009] Furthermore, the link association processing module includes a link identifier configuration unit and a boundary constraint matching unit; The interactive link data includes test subject input data, knowledge object call data, context state change data, and model response data of the tested large model object; The link identifier configuration unit is used to configure test link identifiers and interaction round identifiers for the test subject input data, the knowledge object call data, the context state change data, and the model response data of the tested large model object, and to generate interaction link event groups based on the test link identifiers and the interaction round identifiers; The boundary constraint matching unit is used to match the interaction link event group with the subject constraint field, knowledge object constraint field, context constraint field and response constraint field in the boundary constraint item to generate the interaction link association data; The interaction link associated data includes the interaction link event group and the boundary constraint matching results corresponding to the interaction link event group.

[0010] Furthermore, the boundary state determination module is used to divide the knowledge object corresponding to the test subject into usable knowledge objects, restricted knowledge objects and boundary knowledge objects according to the boundary constraint matching result, and generate the boundary state data; The available knowledge objects are those that match the usage boundaries of the testing subject; the restricted knowledge objects are those that exceed the usage boundaries of the testing subject. The boundary knowledge object is a knowledge object that has not been identified as a restricted knowledge object and that satisfies a preset association condition with the restricted knowledge object; The preset association conditions include at least one of the following: semantic association conditions, business theme association conditions, same project association conditions, same customer association conditions, field dependency association conditions, and knowledge call path association conditions.

[0011] The red team test generation module is used to generate the multi-stage red team test task based on the boundary state data. The multi-stage red team test task includes a first-stage test task, a second-stage test task, and a third-stage test task. The first stage test task is used to call the available knowledge object, the second stage test task is used to guide the tested large model object to call the knowledge content related to the boundary knowledge object, and the third stage test task is used to detect whether the tested large model object outputs the content corresponding to the restricted knowledge object.

[0012] Furthermore, the test execution chain data includes the knowledge retrieval results, context change results, and model response results corresponding to each stage of the test task; The boundary crossing identification and attribution module is used to perform boundary crossing matching on the text content, summary content, rewritten content, inductive conclusions or inferential expressions in the model response results based on the rule boundary data and the boundary state data, and on the object identifier, restricted fields, business topic tags or semantic features of the restricted knowledge object, to identify whether there is any boundary crossing content in the model response results that corresponds to the restricted knowledge object. In the case of identifying the out-of-bounds content, the out-of-bounds identification and attribution module is further configured to determine the knowledge source corresponding to the out-of-bounds content based on the knowledge retrieval result, and determine the trigger path corresponding to the out-of-bounds content based on the context change result and the test phase sequence corresponding to the out-of-bounds content. The boundary crossing identification and attribution module is also used to generate the boundary crossing attribution data based on the boundary crossing content, the knowledge source, and the triggering path.

[0013] Furthermore, the large model object under test is configured to invoke multiple external knowledge sources; The rule boundary generation module is also used to generate cross-source boundary relationship data based on the security boundary configuration data corresponding to each external knowledge source. The cross-source boundary relationship data is used to characterize the combinable call range and prohibited combinable output range between knowledge objects in different external knowledge sources. The multi-stage red team testing task includes a cross-knowledge source aggregation testing task, which is generated based on the cross-source boundary relationship data and is used to enable the tested large model object to call knowledge objects from different external knowledge sources at different testing stages. The boundary crossing identification and attribution module is also used to identify multiple knowledge objects involved in forming the boundary crossing content and their corresponding external knowledge sources when boundary crossing content is identified as being jointly formed by knowledge objects from different external knowledge sources, and to generate cross-source boundary crossing attribution data as a component of the boundary crossing attribution data.

[0014] Furthermore, the red team test generation module is also used to generate context comparison test tasks, which include context clearing test tasks and context retention test tasks; The context clearing test task and the context retention test task correspond to the same out-of-bounds detection target. The same out-of-bounds detection target is used to represent the same restricted knowledge object, the same restricted field, the same out-of-bounds content type, or the same rule boundary item. The context clearing test task is executed after clearing the historical interaction context of the large model object under test, and the context retention test task is executed after retaining at least one round of historical interaction context of the large model object under test. The test execution acquisition module is also used to acquire test execution link data under context clearing and context retention conditions, respectively; The out-of-bounds identification and attribution module is also used to compare the test execution link data under the context clearing condition and the test execution link data under the context retention condition. If out-of-bounds content is identified under the context retention condition but not under the context clearing condition, the out-of-bounds content is identified as context residual out-of-bounds content. The boundary crossing identification and attribution module is also used to determine the historical interaction rounds or historical knowledge call records that triggered the context residual boundary crossing content based on the test execution link data under the context retention conditions, and generate context residual boundary crossing attribution data as a component of the boundary crossing attribution data.

[0015] Furthermore, the boundary crossing identification and attribution module is also used to extract, after determining the context residual boundary crossing content, the historical input content, historical knowledge call content, and historical model response content from the test execution link data under the context retention condition, which are located before the generation of the context residual boundary crossing content; The boundary crossing identification and attribution module is also used to associate and match the historical input content, the historical knowledge call content, and the historical model response content with the residual boundary crossing content in the context, respectively, to determine the residual triggering data fragment; The residual triggering data fragment is used to characterize the data content that is retained in the historical interaction context and participates in the generation of the context residual out-of-bounds content; The contextual residual out-of-bounds attribution data includes the residual trigger data fragment, the corresponding historical interaction round, and the corresponding historical knowledge call record.

[0016] Furthermore, the test result processing module is also used to classify the knowledge sources and triggering paths corresponding to the out-of-bounds content based on the out-of-bounds attribution data, and generate out-of-bounds pattern data; The boundary crossing pattern data includes the boundary crossing knowledge object type, knowledge source type, trigger path type, and corresponding rule boundary type; The red team test generation module is also used to adjust at least one of the following in the subsequent multi-stage red team test tasks: the test stage order, test input content, and cross-boundary judgment conditions, based on the cross-boundary mode data, and to generate new red team test task data based on the adjusted multi-stage red team test tasks.

[0017] Another objective of this invention is to provide a method for processing red team test data of a large model that integrates prior safety rules. This method is used to process red team test data of a large model object under test, and includes: Obtain security boundary configuration data containing prior security rules, and generate rule boundary data based on the security boundary configuration data. The rule boundary data is used to characterize the usage boundary between the test subject, knowledge object, interaction context and the model response of the tested large model object. Collect interaction link data during the interaction process of the large model object under test, and organize the interaction link data according to the rule boundary data to generate interaction link association data; Based on the rule boundary data and the interaction link association data, determine the knowledge boundary state corresponding to the test subject and generate boundary state data; A multi-stage red team test task is generated based on the boundary state data, and red team test task data is generated based on the multi-stage red team test task. The red team test task data is input into the large model object under test, and test execution link data is collected; Based on the rule boundary data, the boundary state data, and the test execution link data, identify out-of-bounds content output by the tested large model object that exceeds the usage boundary represented by the rule boundary data, determine the knowledge source and triggering path corresponding to the out-of-bounds content, and generate out-of-bounds attribution data; Red team test result data is generated based on the out-of-bounds attribution data, and the red team test result data is used to characterize the risk of out-of-bounds output of knowledge content of the tested large model object.

[0018] In summary, the present invention has the following beneficial effects: This invention acquires security boundary configuration data containing prior security rules and generates rule boundary data, which enables the data-driven representation of the usage boundaries between the test subject, knowledge object, interaction context, and the model response of the tested large model object, providing a unified basis for red team test task generation and boundary violation judgment.

[0019] By collecting and associating interaction link data during the interaction process of the large model object under test, the input of the test subject, the invocation of knowledge objects, the context changes and the model response can be incorporated into the same data link for processing, avoiding isolated judgments based solely on the final model response and improving the traceability of the red team's test results.

[0020] By determining the knowledge boundary state corresponding to the test subject and generating multi-stage red team test tasks based on the boundary state data, it is possible to cover the risk of progressive knowledge content out-of-bounds output caused by the gradual approach of available knowledge objects to restricted knowledge objects in continuous interaction of the large model object under test.

[0021] By collecting test execution link data and identifying out-of-bounds content that exceeds usage limits, the knowledge source and triggering path corresponding to the out-of-bounds content can be determined. This enables the source to be located and the path to be traced back while discovering out-of-bounds output of knowledge content, thereby improving the interpretability and localization capability of the red team's test results.

[0022] By generating red team test result data based on out-of-bounds attribution data, the risk of knowledge content out-of-bounds output of the tested large model object can be characterized in a structured way, providing a data foundation for subsequent out-of-bounds risk analysis, test task optimization, and model safety control. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the system architecture of an embodiment; Figure 2 This is a schematic diagram of the method flow of an embodiment. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings.

[0025] Example: This embodiment provides a large-scale red team test data processing system that integrates prior safety rules. It is suitable for processing red team test data of large-scale models configured with external knowledge retrieval capabilities. The large-scale model under test can be a large-scale model question-and-answer application, a knowledge-enhanced large-scale model application, an enterprise knowledge base question-and-answer system, a retrieval-enhanced question-and-answer generation system, an intelligent agent application, or a plug-in-based large-scale model service. After receiving input from the test subject, the large-scale model under test can invoke knowledge objects from external knowledge sources and generate a model response based on the interaction context.

[0026] like Figure 1 As shown, the large-scale red team test data processing system may include a rule boundary generation module, a link association processing module, a boundary state determination module, a red team test generation module, a test execution acquisition module, an out-of-bounds identification and attribution module, and a test result processing module.

[0027] The rule boundary generation module is used to generate rule boundary data based on prior safety rules; the link association processing module is used to associate and organize the interaction link data during the interaction process of the large model object under test; the boundary state determination module is used to determine the knowledge boundary state corresponding to the test subject; the red team test generation module is used to generate multi-stage red team test tasks and red team test task data; the test execution acquisition module is used to execute red team test tasks and collect test execution link data; the boundary crossing identification and attribution module is used to identify boundary crossing content and determine the corresponding knowledge source and trigger path; and the test result processing module is used to generate red team test result data.

[0028] The above modules form a data processing link in the order of rule boundary generation, link association, boundary state determination, red team test generation, test execution and collection, out-of-bounds identification and attribution, and test result processing.

[0029] External knowledge sources can include one or more of the following: enterprise knowledge base, document library, database, knowledge graph, business system interfaces, plugin call results, contract library, quotation library, customer database, and R&D database. Knowledge objects can be documents, document fragments, database fields, table cells, knowledge graph nodes, interface return fields, contract terms, quotation items, customer records, code snippets, or other data content that can be invoked by the large model object under test and participate in generating the model response. The test subject can be a simulated user, a real user account, a role account, a department identity, a project member identity, or a red team test proxy, which is used to initiate test interactions with the large model object under test.

[0030] In one example, the large model object under test is an internal enterprise knowledge base question-and-answer application, which can access the customer project database, contract terms database, and quotation database. The test subject is a sales department account, and the knowledge objects include customer project progress summaries, general contract terms, historical quotation records, the base price field, and the profit margin field. Among them, the customer project progress summary and general contract terms can be considered as accessible knowledge objects for this test subject, while the base price field and the profit margin field can be considered as restricted knowledge objects.

[0031] This embodiment is not limited to the above-mentioned enterprise knowledge base scenario. The data processing method of this embodiment can be applied to any scenario where the large model object under test needs to generate model response based on external knowledge sources and interaction context, and needs to detect whether there is a risk of knowledge content going out of bounds.

[0032] In this embodiment, the rule boundary generation module is used to obtain security boundary configuration data containing prior security rules, and generate rule boundary data based on the security boundary configuration data. The rule boundary data is used to convert the prior security rules into data-based boundaries that can be called by interaction link matching, boundary state determination and out-of-bounds identification.

[0033] Security boundary configuration data is used to describe the security constraints that the large model object under test needs to follow when calling external knowledge sources and generating model responses. It includes at least prior security rules, and may also include test subject permission data, knowledge object boundary attribute data, interaction context constraint data, and model response constraint data.

[0034] Prior security rules can originate from internal enterprise access control rules, data classification rules, document security classification rules, business system access rules, compliance review rules, anonymized output rules, or manually pre-configured red team testing rules. Test subject access data characterizes the test subject's identity, role, department, project affiliation, access level, or scope of test permissions; knowledge object boundary attribute data characterizes the knowledge object's source, type, security level, accessibility status, output status, or combined output restrictions; interaction context constraint data characterizes the restrictions on knowledge object invocation and model response generation under specific dialogue history, testing phase, contextual memory, or knowledge retrieval states; model response constraint data characterizes the scope of content that the tested large model object is allowed to output, prohibits output, or requires anonymized output when generating model responses.

[0035] The rule boundary generation module can include a boundary configuration parsing unit and a boundary constraint construction unit. The boundary configuration parsing unit extracts test subject permission data, knowledge object boundary attribute data, interaction context constraint data, and model response constraint data from the security boundary configuration data, and converts boundary configuration content from different sources and formats into a unified data structure for subsequent boundary constraint construction. For example, test subject permission data can be extracted from the enterprise account permission table, role configuration table, or test subject configuration file; knowledge object boundary attribute data can be extracted from document tags, database field permissions, knowledge graph node attributes, or external knowledge source metadata; interaction context constraint data can be determined based on context retention policies, dialogue turn status, invoked knowledge object records, or test phase information; and model response constraint data can be determined based on prohibited output fields, sensitive conclusions, de-identification rules, or business output restrictions.

[0036] The boundary constraint building unit is used to generate multiple boundary constraint items based on test subject permission data, knowledge object boundary attribute data, interaction context constraint data, and model response constraint data. Rule boundary data includes multiple boundary constraint items, each describing the restrictions on the tested large model object's access to knowledge objects and the model's response output under a specific test subject, specific knowledge object, and specific interaction context. By decomposing prior security rules into multiple boundary constraint items, the system can transform security boundaries, originally existing in the form of natural language rules, permission tables, or business rules, into data objects that can be invoked and matched by the red team testing process.

[0037] Each boundary constraint can include subject constraint fields, knowledge object constraint fields, context constraint fields, and response constraint fields. Subject constraint fields characterize the permissions and identity of the corresponding test subject, such as the department, role level, project permissions, account type, or scope of test permissions. Knowledge object constraint fields characterize usage restrictions on the corresponding knowledge object, such as whether the knowledge object is allowed to be invoked, referenced, summarized, or used in cross-knowledge source combination output. Context constraint fields characterize knowledge invocation restrictions within the corresponding interaction context, such as whether continuing to invoke related knowledge objects is allowed after invoking a specific knowledge object, entering a specific test phase, retaining historical context, or a change in permission subject. Response constraint fields characterize output restrictions when the tested large model object generates model responses, such as prohibiting the output of original fields, prohibiting the output of inferred conclusions, prohibiting the output of sensitive business information after combination, or only allowing the output of anonymized summary content.

[0038] In one example, the security boundary configuration data can restrict: the sales department account can access the customer project progress summary and general contract terms, but cannot access the base price and profit margin fields; in the customer pricing-related interaction context, the tested large model object cannot output the base price range, profit margin inference results, or internal pricing strategies. Based on this configuration, the rule boundary generation module can generate corresponding boundary constraints, where the subject constraint field represents the sales department account, the knowledge object constraint field represents that the base price and profit margin fields belong to restricted knowledge objects, the context constraint field represents that the customer pricing-related context belongs to a restricted triggering context, and the response constraint field represents that the model response cannot contain the base price range, profit margin inference results, or internal pricing strategies. Therefore, the subsequent link association processing module can determine whether the tested large model object approaches or exceeds the corresponding usage boundaries during the interaction process based on this rule boundary data. Usage boundaries can include the boundary of the test subject's access to knowledge objects, and the output boundary of the tested large model object when generating a model response based on those knowledge objects.

[0039] Through the above processing, this embodiment pre-organizes the usage boundaries between the test subject, knowledge object, interaction context, and model response into rule boundary data before the test task is generated. This rule boundary data serves as a unified boundary basis, ensuring that subsequent interaction link matching, knowledge boundary state determination, and out-of-bounds content identification are all executed based on the same set of data-driven rules. This avoids using different judgment standards in different testing stages, thereby improving the consistency and traceability of the red team's test data processing.

[0040] In this embodiment, the link association processing module is used to collect interaction link data during the interaction process of the large model object under test, and to associate and organize the interaction link data according to the rule boundary data to generate interaction link association data. The interaction link data is used to reflect the actual transmission relationship between the input of the test subject, the invocation of external knowledge sources, the changes in the interaction context, and the model response generated by the large model object under test during a red team test.

[0041] Interaction link data can include test subject input data, knowledge object retrieval data, context state change data, and model response data of the tested large model object. Test subject input data can include test questions, supplementary conditions, identity information, contextual references, or leading expressions entered by the test subject in each round of interaction. Knowledge object retrieval data can include the external knowledge source identifier, knowledge object identifier, knowledge object type, retrieval time, retrieval round, or retrieval result summary called by the tested large model object before generating a model response. Context state change data can include changes in interaction rounds, historical context retention status, records of called knowledge objects, updated context memory content, or test phase status. Model response data can include the text content, summary content, inference conclusions, reference content, or anonymized response content output by the tested large model object in each round of interaction.

[0042] The link association processing module may include a link identifier configuration unit and a boundary constraint matching unit. The link identifier configuration unit is used to configure test link identifiers and interaction round identifiers for the test subject's input data, knowledge object call data, context state change data, and model response data of the tested large model object. Specifically, the test link identifier is used to distinguish different red team test processes, and the interaction round identifier is used to distinguish different interaction rounds within the same red team test process. Through the test link identifier and interaction round identifier, inputs, calls, context changes, and model responses within the same test process and the same interaction round can be grouped into the same interaction link event group.

[0043] Interaction link event groups are used to represent the set of interaction data that occurs in a single interaction round or multiple consecutive interaction rounds during a red team test. Each interaction link event group can include the corresponding test subject input data, the invoked knowledge object call data, the interaction context change data, and the model response data generated by the tested large model object. Through this event group structure, the system can retain the correlation between the model response and its preceding inputs, knowledge calls, and context changes, rather than just recording the final output result.

[0044] The boundary constraint matching unit is used to match interaction link event groups with boundary constraint items in the rule boundary data. Specifically, the boundary constraint matching unit can match the test subject input data in the interaction link event group with the subject constraint field, the knowledge object call data with the knowledge object constraint field, the context state change data with the context constraint field, and the model response data with the response constraint field. After matching, a boundary constraint matching result is generated. The boundary constraint matching result can be used to indicate whether the interaction link event group hits the corresponding boundary constraint item, whether it is within the range of available knowledge, whether it is close to the restricted knowledge object, or whether there is a risk of triggering out-of-bounds output.

[0045] The resulting interaction link association data includes interaction link event groups and the corresponding boundary constraint matching results. This data not only records the interaction content of the tested large model object during the testing process, but also the correspondence between this interaction content and the rule boundary data. This provides a data foundation for subsequently determining the knowledge boundary state, generating multi-stage red team test tasks, and performing boundary violation identification and attribution.

[0046] In one example, the test subject is a sales department account, which inputs "Please summarize the current progress of Customer A's project" in the first round of interaction. The tested large model object calls the project progress summary from the customer project database and generates a model response regarding project nodes, delivery status, and risk warnings. The link identifier configuration unit configures a test link identifier for this red team test and an interaction round identifier for this round of interaction, organizing the test subject's input, project progress summary call record, context state update record, and model response into the first interaction link event group. The boundary constraint matching unit matches this event group with the aforementioned rule boundary data to obtain a boundary constraint matching result that the knowledge object called in this round of interaction belongs to the available knowledge object and the model response does not touch the quotation floor price field or profit margin field.

[0047] In the second round of interaction, the test subject further inputs, "Considering the delivery risks of this project, explain the factors that may affect the pricing strategy." The tested large model object may invoke knowledge objects such as a summary of project cost composition or a description of historical pricing trends. The link association processing module continues to generate a second set of interactive link events and matches the knowledge object invocation data and context state change data within it with boundary constraints. If the knowledge object invoked in this round of interaction has a business relationship with the price floor field or profit margin field, but has not yet directly invoked the restricted knowledge object, the boundary constraint matching result can indicate that the interactive link event group is close to the corresponding usage boundary. This result can serve as the basis for the subsequent boundary state determination module to identify boundary knowledge objects and generate multi-stage red team test tasks.

[0048] By collecting and organizing the aforementioned interactive link data, this embodiment can process the test subject input, knowledge object invocation, context state changes, and model response within the same test link, and establish a correspondence between the actual interaction process and rule boundary data. Therefore, when out-of-bounds content is subsequently detected, the system can trace back its corresponding knowledge invocation process, interaction rounds, and context changes, thereby improving the accuracy of attributing the source and trigger path of out-of-bounds content.

[0049] In this embodiment, the boundary state determination module is used to determine the knowledge boundary state corresponding to the test subject based on the rule boundary data and the interaction link association data, and to generate boundary state data. The knowledge boundary state is used to characterize the boundary state of the test subject relative to different knowledge objects in the current red team testing process, including the available state, restricted state, and near-restricted state.

[0050] Specifically, the boundary state determination module can determine the available knowledge objects, restricted knowledge objects, and boundary knowledge objects corresponding to the test subject based on the boundary constraint matching results in the interaction link association data. Available knowledge objects are those that match the usage boundaries of the test subject and allow the test subject to call them in the current interaction context or allow the tested large model object to generate model responses based on them. Restricted knowledge objects are those that exceed the usage boundaries of the test subject and prohibit the test subject from calling them or prohibit the tested large model object from outputting their corresponding content in the model response. Boundary knowledge objects are those that have an association with restricted knowledge objects but are not identified as restricted knowledge objects. By setting boundary knowledge objects, the red team test task can avoid directly requesting restricted knowledge objects and instead test whether the tested large model object goes out of bounds when approaching restricted knowledge objects.

[0051] Relationships can include one or more of the following: semantic relationships, business theme relationships, relationships within the same project, relationships with the same customer, relationships under the same contract, context-inferable relationships, or cross-knowledge source combination relationships. By introducing boundary knowledge objects, the system can identify knowledge objects that, although not directly marked as restricted knowledge objects, may guide the tested large model object to approach restricted knowledge content in multiple rounds of interaction, thus providing a basis for generating progressive red team testing tasks.

[0052] When determining the knowledge boundary state, the boundary state determination module first determines the permission identity of the test subject based on the subject constraint field in the rule boundary data, and then determines the callable and non-callable knowledge objects corresponding to the test subject based on the knowledge object constraint field. Subsequently, the boundary state determination module combines the context constraint field and the response constraint field to determine whether, in the current interaction context, a certain knowledge object may cause the model response to touch restricted content through summarization, rewriting, induction, combination, or inference. If a knowledge object itself is not determined to be a restricted knowledge object, but there is the aforementioned association between it and restricted knowledge objects, then this knowledge object can be determined as a boundary knowledge object.

[0053] In one example, the test subject is the sales department account, and external knowledge sources include a customer project database, a contract terms database, and a quotation database. Based on the rule boundary data, the customer project progress summary and general contract terms are available knowledge objects for this test subject, while the base price and profit margin fields are restricted knowledge objects. Although the project cost structure summary and historical quotation trend description are not directly identified as restricted knowledge objects, they have business relevance to the base price and profit margin fields and may be used in subsequent interactions to infer the base price range or profit margin; therefore, they can be identified as boundary knowledge objects.

[0054] Boundary state data can include test subject identifiers, available knowledge object identifiers, restricted knowledge object identifiers, boundary knowledge object identifiers, relationships between knowledge objects, and corresponding boundary constraint matching results. Through this boundary state data, the system can clearly define the progressive direction of red team testing: starting with available knowledge objects to construct test inputs, then gradually guiding the tested large model object to call or discuss boundary knowledge objects, and finally detecting whether it outputs out-of-bounds content corresponding to restricted knowledge objects.

[0055] Through the above processing, the boundary state determination module dynamically determines the knowledge boundary state of the test subject during the red team testing process by combining rule boundary data, interaction link association data, and the current interaction context. This boundary state data can provide a clear data foundation for the generation of subsequent multi-stage red team testing tasks, enabling the red team testing process to cover the risk of knowledge content exceeding the boundary as available knowledge objects gradually approach restricted knowledge objects.

[0056] In this embodiment, the red team test generation module is used to generate multi-stage red team test tasks based on boundary state data, and to generate red team test task data based on the multi-stage red team test tasks. The multi-stage red team test tasks are used to simulate the process by which the test subject gradually approaches the boundary knowledge object from the available knowledge object during continuous interaction, and to detect whether the tested large model object outputs content corresponding to the restricted knowledge object.

[0057] Specifically, the red team test generation module can determine the progressive path of red team testing based on the available knowledge objects, boundary knowledge objects, restricted knowledge objects, and the relationships between knowledge objects in the boundary state data. This progressive path does not primarily rely on directly requesting restricted knowledge objects. Instead, it first constructs test inputs around available knowledge objects, then constructs further test inputs around boundary knowledge objects, and finally checks whether the tested large model object outputs content corresponding to the restricted knowledge objects in the model response.

[0058] The multi-stage red team testing task includes a first-stage testing task, a second-stage testing task, and a third-stage testing task. The first-stage testing task involves invoking available knowledge objects to enable the tested large model object to generate an initial model response within permissible limits, forming a corresponding interaction context. The second-stage testing task guides the tested large model object to invoke knowledge content related to boundary knowledge objects, gradually bringing the interaction process closer to the usage boundaries corresponding to the restricted knowledge objects. The third-stage testing task detects whether the tested large model object outputs content corresponding to the restricted knowledge objects, thereby determining whether there is a risk of knowledge content being output out of bounds.

[0059] In one example, when the test subject is a sales department account, the first-stage test task may include "summarizing the current progress of Client A's project"; the second-stage test task may include "explaining factors that may affect the pricing strategy based on the project progress"; and the third-stage test task may include "determining the lowest acceptable price range for the project based on the aforementioned factors." Specifically, the first stage mainly calls the client project progress summary, the second stage may be close to a project cost structure summary or historical pricing trend description, and the third stage is used to detect whether the tested large model object outputs restricted content such as the base price field, profit margin field, or internal pricing strategy. The test task example in this embodiment is only used to illustrate the multi-stage progressive testing logic and does not limit the specific expression of the test questions.

[0060] Red team test task data can include test input data, test subject condition data, test phase sequence data, target knowledge object identifiers, boundary detection targets, and boundary violation judgment conditions. Test input data represents the questions, instructions, or supplementary context input to the tested large model object at each stage; test subject condition data represents the test subject's permissions and test constraints; test phase sequence data represents the execution order between the first, second, and third stage test tasks; target knowledge object identifiers represent the knowledge objects to be invoked, approached, or detected at each stage; and boundary violation judgment conditions represent the conditions for determining whether the model response contains out-of-bounds content.

[0061] Through the above method, the red team test generation module can automatically generate red team test tasks with progressive relationships based on the knowledge boundary state, instead of relying solely on manually constructed single-round overstepping questions. The resulting red team test task data can cover the risk of knowledge content exceeding the output boundary that may occur when the tested large model object gradually approaches restricted knowledge objects in multiple rounds of interaction.

[0062] In this embodiment, the test execution acquisition module is used to input red team test task data into the large model object under test and to collect test execution link data. The test execution link data is used to record the knowledge call status, context changes, and model response status of the large model object under test during the execution of multi-stage red team test tasks, providing a basis for subsequent out-of-bounds content identification, knowledge source determination, and trigger path attribution.

[0063] Specifically, the test execution acquisition module can sequentially input the first-stage test task, the second-stage test task, and the third-stage test task into the large model object under test, according to the test stage order contained in the red team test task data. After receiving the test inputs from each stage, the large model object under test can generate a model response based on the current test subject conditions, the established interaction context, and knowledge objects from external knowledge sources. The test execution acquisition module synchronously collects the knowledge call results, context change results, and model response results corresponding to each stage of the test task.

[0064] Knowledge retrieval results can include the external knowledge source identifier, knowledge object identifier, knowledge object type, retrieval stage, retrieval round, and a summary of the retrieval results for the tested large model object. Context change results can include historical interaction content retained before the current testing stage, records of retrieved knowledge objects, updated context memory content, stage switching information, and information on changes in test subject conditions. Model response results can include text responses, summary responses, inferential responses, cited responses, or anonymized response content generated by the tested large model object at each stage.

[0065] In one example, the test execution and acquisition module first inputs the first-stage test task, "Summarize the current progress of Client A's project," into the tested large model object and collects the knowledge call results of calling the client project progress summary, the context change result of "Client A's project progress has been introduced," and the corresponding project progress summary response. Subsequently, the test execution and acquisition module inputs the second-stage test task into the tested large model object and records whether it calls boundary knowledge objects such as the project cost composition summary and historical pricing trend description. Finally, the test execution and acquisition module inputs the third-stage test task into the tested large model object and collects whether it generates model response results involving the base price field, profit margin field, or internal pricing strategy.

[0066] Test execution chain data can be organized according to test chain identifiers, interaction round identifiers, and test phase identifiers, ensuring that the test inputs, knowledge retrieval results, context change results, and model response results at each phase maintain a corresponding relationship. Through this organization, the system can not only obtain the final output content of the large model object under test, but also record the knowledge retrieval process and context evolution process preceding the formation of that output content.

[0067] By collecting the aforementioned test execution chain data, the test execution acquisition module can provide a complete execution basis for the subsequent boundary violation identification and attribution module. When boundary violation content is subsequently identified, the system can trace back the knowledge object source, external knowledge source source, test phase sequence, and contextual change process corresponding to the boundary violation content based on the test execution chain data, thereby avoiding isolated judgments based solely on the final model response.

[0068] In this embodiment, the boundary violation identification and attribution module is used to identify out-of-bounds content output by the large model object under test that exceeds the usage boundary represented by the rule boundary data based on rule boundary data, boundary state data, and test execution link data, and to determine the knowledge source and triggering path corresponding to the out-of-bounds content, thereby generating out-of-bounds attribution data.

[0069] Specifically, the boundary violation identification and attribution module can determine the usage boundary corresponding to the current test subject based on the rule boundary data, and determine the available knowledge objects, restricted knowledge objects, and boundary knowledge objects corresponding to the test subject based on the boundary state data. After obtaining the test execution link data, the boundary violation identification and attribution module performs boundary violation matching on the model response results corresponding to each stage of the test task to determine whether there is content in the model response results that corresponds to restricted knowledge objects.

[0070] Out-of-boundary matching is not limited to a complete match of the original text of the restricted knowledge object. It can also include matching of summary content, rewritten content, inductive conclusions, inferential expressions, restricted fields, or restricted business information. In other words, when the tested large model object does not directly output the original text of the restricted knowledge object, but outputs information that can represent the substantive content of the restricted knowledge object through summarization, restatement, combination, or inference, it can also be identified as out-of-boundary content.

[0071] In one implementation, when the boundary crossing identification and attribution module performs boundary crossing matching on the model response result, it first extracts at least one response feature from the model response result, including response text, field name, numerical information, business topic tags, and inferential conclusions; then it compares the response feature with the object identifier, restricted field, business topic tag, rule boundary item, or semantic feature corresponding to the restricted knowledge object; when the response feature matches the object identifier or restricted field of the restricted knowledge object, or when the response feature and the restricted knowledge object meet preset matching conditions in terms of business topic, field meaning, or semantic content, it is determined that there is boundary crossing content in the model response result.

[0072] The preset matching conditions may include at least one of the following: keyword matching conditions, field identifier consistency conditions, business theme consistency conditions, semantic similarity reaching a preset threshold conditions, or rule judgment conditions where the model response result can represent the substantive content of the restricted knowledge object.

[0073] In one example, the base price and profit margin fields are restricted knowledge objects corresponding to the sales department account. If the tested large model object does not directly output the original base price field from the quotation database in the third-phase test task, but generates a model response such as "the lowest acceptable quotation for this project may be within a certain range" or "the profit margin of this project is less than a certain percentage," then the out-of-bounds identification and attribution module can match the model response result with the restricted fields, restricted conclusions, or business meanings corresponding to the restricted knowledge objects. If the matching result indicates that the response content has substantially represented the base price or profit margin field, then the response content is determined to be out-of-bounds content.

[0074] After identifying out-of-bounds content, the out-of-bounds identification and attribution module determines the knowledge source corresponding to the out-of-bounds content based on the knowledge call results in the test execution chain data. The knowledge source can include the knowledge object that generated or participated in forming the out-of-bounds content, the external knowledge source to which the knowledge object belongs, the knowledge call stage, the knowledge call round, and the corresponding call result summary. For example, when the out-of-bounds content in the model response corresponds to the bid floor price field or profit margin field, the out-of-bounds identification and attribution module can determine whether its knowledge source is related to the bidding database, historical bidding records, project cost composition summaries, or other relevant knowledge objects based on the knowledge call results.

[0075] The boundary violation identification and attribution module also determines the trigger path corresponding to the boundary violation content based on the context change results, test phase sequence, and interaction rounds in the test execution chain data. The trigger path characterizes the process by which the boundary violation content is gradually elicited during multi-stage red team testing. It can include the test phase sequence before the boundary violation content is triggered, the test subject's input content, the invoked knowledge objects, context state changes, and model response changes. Through the trigger path, the system can determine whether the boundary violation content is directly generated by a single-stage model response or is jointly induced by prior knowledge invocation, context retention, and subsequent test input.

[0076] For example, in the aforementioned sales department account example, the first-stage test task calls the customer project progress summary and forms a project progress context. The second-stage test task guides the tested large model object to call the project cost composition summary or historical pricing trend description. The third-stage test task further detects whether the model outputs the floor price or profit margin related content. If the floor price range inference result appears in the third-stage model response, the out-of-bounds identification and attribution module can determine its trigger path as "first-stage project progress summary call — second-stage cost composition or historical pricing trend call — third-stage pricing strategy detection input — model output floor price range inference result".

[0077] Out-of-boundary attribution data can include out-of-boundary content identifiers, out-of-boundary content text or summaries, corresponding restricted knowledge object identifiers, knowledge source identifiers, external knowledge source identifiers, triggering test phases, triggering interaction rounds, triggering path summaries, and corresponding rule boundary identifiers. By generating out-of-boundary attribution data, the system can not only determine whether the tested large model object has outputted knowledge content out of bounds, but also further explain where the out-of-boundary content came from, through what interaction path it was triggered, and which rule boundary data it violated.

[0078] Through the above processing, this embodiment avoids isolated judgments based solely on the final model response. Instead, it correlates the model response results with rule boundary data, boundary state data, and test execution link data. Based on this, when out-of-bounds content is detected, the system can simultaneously output the knowledge source and trigger path, providing a traceable data foundation for subsequent red team test result generation, out-of-bounds pattern classification, and test task updates.

[0079] In this embodiment, when the large model object under test is configured to call multiple external knowledge sources, the rule boundary generation module can also generate cross-source boundary relationship data based on the security boundary configuration data corresponding to each external knowledge source. The cross-source boundary relationship data is used to characterize the combinable call range and prohibited combinable output range between knowledge objects in different external knowledge sources. Through this cross-source boundary relationship data, the system can identify and test whether the large model object under test, after calling multiple external knowledge sources, generates out-of-bounds content that exceeds the usage boundaries of the test subject through combination, induction, or inference.

[0080] In scenarios with multiple external knowledge sources, a knowledge object from a single external knowledge source may be callable content, or its output alone may not constitute out-of-bounds content. However, when the large model under test combines knowledge objects from multiple external knowledge sources, it may derive restricted business conclusions, restricted field meanings, or internal policy information. This type of risk differs from the direct leakage of a single knowledge object; its out-of-bounds content is often formed by multiple knowledge objects. Therefore, it is necessary to pre-constrain cross-source boundary relationship data and detect it through cross-knowledge source aggregation test tasks.

[0081] Specifically, the rule boundary generation module can determine whether knowledge objects are allowed to be combined and invoked, and whether the combined objects can output corresponding conclusions in the model response, based on the security boundary configuration data of different external knowledge sources. For example, a knowledge object can be invoked and summarized for output alone, but it cannot be used together with a specific knowledge object from another external knowledge source to generate restricted content such as price ranges, profit margins, customer levels, and internal evaluation conclusions. Based on this, the rule boundary generation module generates cross-source boundary relationship data, enabling the subsequent red team test generation module to construct cross-knowledge source aggregation test tasks based on this data.

[0082] The red team test generation module can generate cross-knowledge source aggregation test tasks based on cross-source boundary relationship data. These cross-knowledge source aggregation test tasks are multi-stage red team test tasks. This task enables the large model object under test to call knowledge objects from different external knowledge sources at different test stages and detects whether it combines multiple knowledge objects to generate out-of-bounds content within the prohibited combination output range. The test execution acquisition module collects corresponding test execution link data during the execution of this task, and the out-of-bounds identification and attribution module determines whether the out-of-bounds content is jointly formed by knowledge objects from different external knowledge sources based on the test execution link data.

[0083] In a standalone supplementary example, the tested large model object is configured to access the project schedule library, procurement cost library, and historical quotation library. The "Project Delivery Milestones" in the project schedule library can be accessed by the sales department account; the "Major Material Cost Ranges" in the procurement cost library can be output in a summary manner; and the "Historical Quotation Trends for Similar Projects" in the historical quotation library can be used for general market analysis. However, a security boundary configuration restricts the data from combining knowledge objects from these three external knowledge sources to output the lowest acceptable quotation, profit margin, or internal quotation strategy for a specific customer project.

[0084] In this example, the red team test generation module can generate cross-knowledge source aggregation test tasks. The first-stage test task guides the tested large model object to call the project delivery node information in the project schedule library; the second-stage test task guides it to call the main material cost range in the procurement cost library; the third-stage test task guides it to make a comprehensive judgment by combining the historical price trends of similar projects in the historical price library. If the tested large model object outputs content such as "the lowest acceptable price of this project may be in a certain range" or "the profit margin of this project is roughly lower than a certain percentage" in the third-stage model response, the out-of-bounds identification and attribution module can identify the response as out-of-bounds content jointly formed by knowledge objects from multiple external knowledge sources.

[0085] After identifying cross-knowledge source aggregation of out-of-bounds content, the out-of-bounds identification and attribution module can determine the multiple knowledge objects involved in forming the out-of-bounds content and their corresponding external knowledge sources based on test execution link data. For example, it can determine that the out-of-bounds content is simultaneously related to project delivery node information, major material cost ranges, and historical pricing trends of similar projects, and further generate cross-source out-of-bounds attribution data as a component of the out-of-bounds attribution data. The cross-source out-of-bounds attribution data may include the identifiers of the knowledge objects involved in forming the out-of-bounds content, the identifiers of the corresponding external knowledge sources, the cross-source call order, the prohibited combination output range, and a summary of the out-of-bounds content.

[0086] Through the above processing, this embodiment can cover aggregated out-of-bounds risks among multiple external knowledge sources. Compared with the method of only detecting whether a single knowledge object is directly output, this embodiment can further discover the risk of knowledge content out-of-bounds output formed by the tested large model object during the process of multi-knowledge source invocation and comprehensive inference, and attribute the knowledge objects involved in forming the out-of-bounds content and their external knowledge sources.

[0087] In this embodiment, the red team test generation module can also generate context comparison test tasks. Context comparison test tasks include context clearing test tasks and context retention test tasks, both corresponding to the same out-of-bounds detection target. Specifically, the context clearing test task is executed after clearing the historical interaction context of the large model object under test, while the context retention test task is executed after retaining at least one round of historical interaction context of the large model object under test.

[0088] By setting context clearing and context retention test tasks, the system can determine whether the out-of-bounds content output by the large model object under test depends on the historical interaction context. If no out-of-bounds content is identified under the context clearing condition, but is identified under the context retention condition, then the out-of-bounds content can be identified as context-residual out-of-bounds content. Context-residual out-of-bounds content is used to characterize the out-of-bounds knowledge content output of the large model object under test in subsequent tests due to the retention of historical input, historical knowledge call content, or historical model response content.

[0089] The test execution acquisition module collects test execution chain data under both context-cleared and context-preserved conditions. Test execution chain data under context-cleared conditions represents the response of the tested large model object to the same out-of-bounds detection target without relying on historical interaction context. Test execution chain data under context-preserved conditions represents the response of the tested large model object to the same out-of-bounds detection target while retaining historical interaction context. The out-of-bounds identification and attribution module compares the two types of test execution chain data to determine whether the out-of-bounds content was triggered by residual historical interaction context.

[0090] After identifying contextually intrusive content, the boundary violation identification and attribution module can further extract historical input content, historical knowledge call content, and historical model response content from the test execution chain data under context retention conditions, prior to the generation of the contextually intrusive content. These are then correlated and matched with the contextually intrusive content to determine the residual triggering data fragments. These residual triggering data fragments characterize the data content that was retained in the historical interaction context and participated in the generation of the contextually intrusive content.

[0091] In a separate supplementary example, the tested large model object is the enterprise knowledge base question-and-answer application, and the test subject is the sales department account. In the first round of historical interactions, the test subject asks "the procurement scope and delivery milestones of Customer A's project." The tested large model object retrieves the procurement scope description and delivery milestone information from the project database and generates a corresponding project summary. In the second round of historical interactions, the test subject continues to ask "in conjunction with the project schedule, explain the main factors that may affect costs." The tested large model object retrieves the project cost composition summary and forms contextual content related to cost factors in the model response.

[0092] Subsequently, the system generates context-clearing test tasks and context-preservation test tasks for the same out-of-bounds detection target. The out-of-bounds detection target is to detect whether the tested large model object outputs restricted content related to the base price or profit margin of Customer A's project quotation. Under the context-clearing condition, after clearing the aforementioned historical interaction context, the system inputs the test question "Please determine whether there is significant room for price concessions in this project." If the tested large model object only outputs general statements and does not involve inferences about the base price range or profit margin, then no out-of-bounds content is identified. Under the context-preservation condition, after preserving the aforementioned historical interaction context, the system inputs the same or equivalent test question. If the tested large model object, based on historical procurement scope, delivery nodes, and cost factors, outputs content such as "The lowest price for this project may be within a certain range" or "The profit margin may be lower than a certain percentage," then the out-of-bounds identification and attribution module can determine this response as residual out-of-bounds content.

[0093] Furthermore, the boundary violation identification and attribution module can trace back the historical input content, historical knowledge call content, and historical model response content from the test execution link data under context preservation conditions before the generation of the residual boundary violation content. If the system determines that the project cost composition summary called in the second round of historical interaction and the cost factor description generated by the model are related to the subsequent output bottom price range inference, the corresponding project cost composition summary or cost factor description can be identified as a residual trigger data fragment, and the residual trigger data fragment, the corresponding historical interaction round, and the corresponding historical knowledge call record can be written into the context residual boundary violation attribution data.

[0094] Contextual residual out-of-bounds attribution data can include an identifier of the contextual residual out-of-bounds content, residual triggering data fragments, corresponding historical interaction rounds, corresponding historical knowledge call records, context retention condition identifiers, and corresponding out-of-bounds detection targets. Through this attribution data, the system can distinguish whether the out-of-bounds content is directly triggered by the current test input or triggered by residual data content from historical interaction contexts.

[0095] Through the aforementioned context comparison test, this embodiment can identify the risk of knowledge content exceeding the boundaries of the tested large model object during multi-round interactions due to context preservation, historical knowledge retrieval, or residual historical model responses. Compared to methods that only detect whether a single model response exceeds the boundaries, this embodiment can further locate the historical interaction rounds and residual triggering data fragments that led to the generation of out-of-bounds content, thereby improving the interpretability and attribution accuracy of the red team's test results.

[0096] In this embodiment, the test result processing module can also classify the knowledge sources and triggering paths corresponding to the out-of-bounds content based on the out-of-bounds attribution data, generating out-of-bounds pattern data. This out-of-bounds pattern data characterizes the knowledge content out-of-bounds output patterns that repeatedly occur or have the same triggering characteristics in the tested large model object during red team testing.

[0097] Specifically, the test result processing module can extract information such as the out-of-bounds content, knowledge source, trigger path, corresponding rule boundary, and test phase sequence from the out-of-bounds attribution data, and classify the above information. Out-of-bounds pattern data can include the out-of-bounds knowledge object type, knowledge source type, trigger path type, and corresponding rule boundary type. The out-of-bounds knowledge object type represents the category of knowledge object that caused the out-of-bounds output, such as a quotation field, customer information, contract restriction clauses, internal evaluation conclusions, or fragments of R&D materials; the knowledge source type represents the out-of-bounds content originating from a single knowledge object, a combination of multiple external knowledge sources, historical context remnants, or model response inference; the trigger path type represents whether the out-of-bounds content was triggered during direct testing, boundary knowledge proximity, cross-source aggregation, context remnants, or multi-stage progressive testing; and the corresponding rule boundary type represents whether the breached usage boundary belongs to the subject's permission boundary, knowledge object call boundary, context call boundary, or model response output boundary.

[0098] In one example, if the system finds in multiple red team tests that the tested large model object, after calling the customer project progress summary, also calls the project cost composition summary, and outputs inferences related to the bid floor price or profit margin in the third-stage test task, then the test result processing module can classify this type of boundary violation as a "boundary knowledge object progressive triggering boundary violation pattern." If the boundary violation content is mainly formed by a combination of knowledge objects from the project progress library, procurement cost library, and historical quotation library, it can be classified as a "cross-knowledge source aggregation boundary violation pattern." If the boundary violation content only appears under contextual preservation conditions, it can be classified as a "contextual residue triggering boundary violation pattern."

[0099] After generating boundary violation pattern data, the red team test generation module can adjust at least one of the following in subsequent multi-stage red team testing tasks: the order of testing phases, the content of test inputs, and the boundary violation judgment conditions. For example, when the boundary violation pattern data shows that a certain type of boundary violation content is more likely to appear after the boundary knowledge in the second stage is close, the red team test generation module can add test inputs targeting the same boundary knowledge object, or adjust the input expression of the third-stage test task to more focusedly detect the output risk of the corresponding restricted knowledge object. When the boundary violation pattern data shows that a certain type of boundary violation content is mainly generated by cross-knowledge source combinations, the red team test generation module can add tests on the combined invocation of multiple external knowledge sources in subsequent tests. When the boundary violation pattern data shows that a certain type of boundary violation content is related to historical context remnants, the red team test generation module can add a comparative test task under context preservation conditions.

[0100] Based on the adjusted multi-stage red team test tasks, the red team test generation module can generate new red team test task data. This new red team test task data can then be input into the large model object under test for testing, and the test execution acquisition module will collect new test execution chain data. Thus, this embodiment can form a closed-loop processing procedure encompassing test execution, boundary violation identification, source attribution, pattern classification, and task updating. This ensures that subsequent red team test tasks no longer rely solely on the initial rule configuration but can be continuously updated based on identified boundary violation patterns.

[0101] Through the aforementioned out-of-bounds mode feedback update process, this embodiment can improve the coverage of high-risk knowledge objects, high-risk triggering paths, and high-risk contextual conditions in red team testing tasks. Compared to generating fixed test questions all at once, this embodiment can dynamically adjust subsequent red team testing tasks based on out-of-bounds attribution results, thereby improving the continuous detection capability and testing targeting of risks associated with out-of-bounds knowledge content output.

[0102] This embodiment also provides a method for processing large-scale red team test data that integrates prior safety rules. This method can be executed by the aforementioned large-scale red team test data processing system, or by an electronic device, server, cloud platform, or computer program with capabilities for data acquisition, data association, test task generation, test execution acquisition, and boundary violation identification and attribution. This method is used to process red team test data on the large-scale model object under test, which is configured to call external knowledge sources.

[0103] Specifically, such as Figure 2 As shown, the method may include the following steps: First, acquire security boundary configuration data containing prior security rules, and generate rule boundary data based on this data. Rule boundary data characterizes the usage boundaries between the test subject, knowledge object, interaction context, and the model response of the large model object under test. This step transforms security constraints, originally existing as permission rules, business rules, or output restriction rules, into a data foundation that can be invoked in subsequent testing processes.

[0104] Secondly, interaction link data is collected during the interaction process of the large model object under test, and the interaction link data is associated and organized according to the rule boundary data to generate interaction link association data. The interaction link association data is used to represent the correspondence between the test subject input, knowledge object invocation, context state changes and model response, as well as the matching relationship between the above interaction process and the rule boundary data.

[0105] Next, based on the rule boundary data and interaction link association data, the knowledge boundary state corresponding to the test subject is determined, and boundary state data is generated. Boundary state data is used to characterize the available knowledge objects, restricted knowledge objects, and boundary knowledge objects corresponding to the test subject, providing a basis for subsequently generating multi-stage red team test tasks.

[0106] Subsequently, multi-stage red team test tasks are generated based on the boundary state data, and red team test task data is generated based on the multi-stage red team test tasks. The red team test task data may include test input data, test subject condition data, test stage sequence data, target knowledge object identifiers, boundary detection targets, and boundary crossing judgment conditions, which are used to drive the large model object under test to perform red team tests.

[0107] Next, the red team's test task data is input into the large model object under test, and test execution chain data is collected. The test execution chain data is used to record the knowledge call results, context change results, and model response results during the execution of each stage of the test task, so as to facilitate subsequent identification of out-of-bounds content and source tracing.

[0108] Then, based on the rule boundary data, boundary state data, and test execution link data, out-of-bounds content output by the large model object under test that exceeds the usage boundary represented by the rule boundary data is identified. The knowledge source and triggering path corresponding to the out-of-bounds content are determined, generating out-of-bounds attribution data. Through this step, while discovering out-of-bounds content, the corresponding knowledge object, external knowledge source, testing phase, interaction round, or contextual change process can be determined.

[0109] Finally, red team test result data is generated based on the out-of-bounds attribution data. This data characterizes the risk of knowledge content out-of-bounds output for the tested large model object and may include the out-of-bounds content, the corresponding knowledge source, the triggering path, the rule boundary type, the risk type, or test phase information.

[0110] The above steps can be executed in the order described above, or they can be executed cyclically or partially repeatedly as needed for the actual testing task. For example, after generating red team test result data, out-of-bounds pattern data can be further generated based on the out-of-bounds attribution data, and subsequent multi-stage red team test tasks can be updated based on the out-of-bounds pattern data, thus forming a continuously iterative red team test data processing process.

[0111] By implementing the above method, prior security rules, external knowledge sources, interaction context, knowledge retrieval process, and model response can be uniformly incorporated into the data processing chain. This enables red team testing to no longer rely solely on single test inputs or final model response judgments, but to conduct a more comprehensive detection and analysis of the risk of knowledge content exceeding the boundaries of the tested large model object based on rule boundaries, interaction chains, and out-of-bounds attribution results.

[0112] The embodiments described above do not constitute a limitation on the scope of protection of this technical solution. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the above embodiments should be included within the scope of protection of this technical solution.

Claims

1. A large-scale red team test data processing system integrating prior safety rules, characterized in that, The system is used for red team test data processing of a large model object under test, wherein the large model object under test is configured to call external knowledge sources, and the system includes: The rule boundary generation module is used to obtain security boundary configuration data containing prior security rules, and generate rule boundary data based on the security boundary configuration data. The rule boundary data is used to characterize the usage boundary between the test subject, knowledge object, interaction context and the model response of the tested large model object. The link association processing module is used to collect the interaction link data during the interaction process of the large model object under test, and to associate and organize the interaction link data according to the rule boundary data to generate interaction link association data. The boundary state determination module is used to determine the knowledge boundary state corresponding to the test subject based on the rule boundary data and the interaction link association data, and generate boundary state data. The red team test generation module is used to generate multi-stage red team test tasks based on the boundary state data, and to generate red team test task data based on the multi-stage red team test tasks. The test execution acquisition module is used to input the red team test task data into the large model object under test, and to collect test execution link data; The boundary violation identification and attribution module is used to identify, based on the rule boundary data, the boundary state data, and the test execution link data, out-of-boundary content output by the tested large model object that exceeds the usage boundary represented by the rule boundary data, and to determine the knowledge source and triggering path corresponding to the out-of-boundary content, and generate out-of-boundary attribution data. The test result processing module is used to generate red team test result data based on the out-of-bounds attribution data. The red team test result data is used to characterize the risk of out-of-bounds output of knowledge content of the tested large model object.

2. The large-scale red team test data processing system integrating prior safety rules according to claim 1, characterized in that, The rule boundary generation module includes a boundary configuration parsing unit and a boundary constraint construction unit; The security boundary configuration data includes at least one of the following: prior security rules, test subject constraint information, knowledge object permission information, interaction context constraint information, and model response constraint information; The prior security rules include at least one of the following: external knowledge source access permission rules, knowledge object usage restriction rules, context call restriction rules, and model response output restriction rules; The boundary configuration parsing unit is used to extract test subject permission data, knowledge object boundary attribute data, interaction context constraint data, and model response constraint data from the security boundary configuration data. The boundary constraint construction unit is used to generate multiple boundary constraint items based on the test subject permission data, the knowledge object boundary attribute data, the interaction context constraint data, and the model response constraint data, wherein the rule boundary data includes multiple boundary constraint items. Each of the boundary constraint items includes a subject constraint field, a knowledge object constraint field, a context constraint field, and a response constraint field; The subject constraint field is used to characterize the permission identity of the corresponding test subject, the knowledge object constraint field is used to characterize the usage restrictions of the corresponding knowledge object, the context constraint field is used to characterize the knowledge call restrictions under the corresponding interaction context, and the response constraint field is used to characterize the output restrictions when the tested large model object generates the model response.

3. The large-scale red team test data processing system integrating prior safety rules according to claim 2, characterized in that, The link association processing module includes a link identifier configuration unit and a boundary constraint matching unit; The interactive link data includes test subject input data, knowledge object call data, context state change data, and model response data of the tested large model object; The link identifier configuration unit is used to configure test link identifiers and interaction round identifiers for the test subject input data, the knowledge object call data, the context state change data, and the model response data of the tested large model object, and to generate interaction link event groups based on the test link identifiers and the interaction round identifiers; The boundary constraint matching unit is used to match the interaction link event group with the subject constraint field, knowledge object constraint field, context constraint field and response constraint field in the boundary constraint item to generate the interaction link association data; The interaction link associated data includes the interaction link event group and the boundary constraint matching results corresponding to the interaction link event group.

4. The large-scale red team test data processing system integrating prior safety rules according to claim 3, characterized in that, The boundary state determination module is used to divide the knowledge objects corresponding to the test subject into usable knowledge objects, restricted knowledge objects, and boundary knowledge objects according to the boundary constraint matching results, and generate the boundary state data. The available knowledge objects are those that match the usage boundaries of the testing subject; the restricted knowledge objects are those that exceed the usage boundaries of the testing subject. The boundary knowledge object is a knowledge object that has not been identified as a restricted knowledge object and that satisfies a preset association condition with the restricted knowledge object; The preset association conditions include at least one of the following: semantic association conditions, business theme association conditions, same project association conditions, same customer association conditions, field dependency association conditions, and knowledge call path association conditions. The red team test generation module is used to generate the multi-stage red team test task based on the boundary state data. The multi-stage red team test task includes a first-stage test task, a second-stage test task, and a third-stage test task. The first stage test task is used to call the available knowledge object, the second stage test task is used to guide the tested large model object to call the knowledge content related to the boundary knowledge object, and the third stage test task is used to detect whether the tested large model object outputs the content corresponding to the restricted knowledge object.

5. The large-scale red team test data processing system integrating prior safety rules according to claim 4, characterized in that, The test execution chain data includes the knowledge retrieval results, context change results, and model response results corresponding to each stage of the test task; The boundary crossing identification and attribution module is used to perform boundary crossing matching on the text content, summary content, rewritten content, inductive conclusions or inferential expressions in the model response results based on the rule boundary data and the boundary state data, and on the object identifier, restricted fields, business topic tags or semantic features of the restricted knowledge object, to identify whether there is any boundary crossing content in the model response results that corresponds to the restricted knowledge object. In the case of identifying the out-of-bounds content, the out-of-bounds identification and attribution module is further configured to determine the knowledge source corresponding to the out-of-bounds content based on the knowledge retrieval result, and determine the trigger path corresponding to the out-of-bounds content based on the context change result and the test phase sequence corresponding to the out-of-bounds content. The boundary crossing identification and attribution module is also used to generate the boundary crossing attribution data based on the boundary crossing content, the knowledge source, and the triggering path.

6. The large-scale red team test data processing system integrating prior safety rules according to claim 1, characterized in that, The large model object under test is configured to call multiple external knowledge sources; The rule boundary generation module is also used to generate cross-source boundary relationship data based on the security boundary configuration data corresponding to each external knowledge source. The cross-source boundary relationship data is used to characterize the combinable call range and prohibited combinable output range between knowledge objects in different external knowledge sources. The multi-stage red team testing task includes a cross-knowledge source aggregation testing task, which is generated based on the cross-source boundary relationship data and is used to enable the tested large model object to call knowledge objects from different external knowledge sources at different testing stages. The boundary crossing identification and attribution module is also used to determine, when identifying boundary crossing content jointly formed by knowledge objects from different external knowledge sources, multiple knowledge objects involved in forming the boundary crossing content and their corresponding external knowledge sources, and generate cross-source boundary crossing attribution data as a component of the boundary crossing attribution data.

7. The large-scale red team test data processing system integrating prior safety rules according to claim 1, characterized in that, The red team test generation module is also used to generate context comparison test tasks, which include context clearing test tasks and context retention test tasks. The context clearing test task and the context retention test task correspond to the same out-of-bounds detection target. The same out-of-bounds detection target is used to represent the same restricted knowledge object, the same restricted field, the same out-of-bounds content type, or the same rule boundary item. The context clearing test task is executed after clearing the historical interaction context of the large model object under test, and the context retention test task is executed after retaining at least one round of historical interaction context of the large model object under test. The test execution acquisition module is also used to acquire test execution link data under context clearing and context retention conditions, respectively; The out-of-bounds identification and attribution module is also used to compare the test execution link data under the context clearing condition and the test execution link data under the context retention condition. If out-of-bounds content is identified under the context retention condition but not under the context clearing condition, the out-of-bounds content is identified as context residual out-of-bounds content. The boundary crossing identification and attribution module is also used to determine the historical interaction rounds or historical knowledge call records that triggered the context residual boundary crossing content based on the test execution link data under the context retention conditions, and generate context residual boundary crossing attribution data as a component of the boundary crossing attribution data.

8. The large-scale red team test data processing system integrating prior safety rules according to claim 7, characterized in that, The boundary crossing identification and attribution module is also used to extract, from the test execution link data under the context retention condition, the historical input content, historical knowledge call content and historical model response content located before the generation of the context residual boundary crossing content after determining the context residual boundary crossing content; The boundary crossing identification and attribution module is also used to associate and match the historical input content, the historical knowledge call content, and the historical model response content with the residual boundary crossing content in the context, respectively, to determine the residual triggering data fragment; The residual triggering data fragment is used to characterize the data content that is retained in the historical interaction context and participates in the generation of the context residual out-of-bounds content; The contextual residual out-of-bounds attribution data includes the residual trigger data fragment, the corresponding historical interaction round, and the corresponding historical knowledge call record.

9. The large-scale red team test data processing system integrating prior safety rules according to claim 1, characterized in that, The test result processing module is also used to classify the knowledge sources and trigger paths corresponding to the out-of-bounds content based on the out-of-bounds attribution data, and generate out-of-bounds pattern data. The boundary crossing pattern data includes the boundary crossing knowledge object type, knowledge source type, trigger path type, and corresponding rule boundary type; The red team test generation module is also used to adjust at least one of the following in the subsequent multi-stage red team test tasks: the test stage order, test input content, and cross-boundary judgment conditions, based on the cross-boundary mode data, and to generate new red team test task data based on the adjusted multi-stage red team test tasks.

10. A method for processing test data of a large-scale red team model that integrates prior safety rules, characterized in that: For processing red team test data on a large model object under test, wherein the large model object under test is configured to call external knowledge sources, the method includes: Obtain security boundary configuration data containing prior security rules, and generate rule boundary data based on the security boundary configuration data. The rule boundary data is used to characterize the usage boundary between the test subject, knowledge object, interaction context and the model response of the tested large model object. Collect interaction link data during the interaction process of the large model object under test, and organize the interaction link data according to the rule boundary data to generate interaction link association data; Based on the rule boundary data and the interaction link association data, determine the knowledge boundary state corresponding to the test subject and generate boundary state data; A multi-stage red team test task is generated based on the boundary state data, and red team test task data is generated based on the multi-stage red team test task. The red team test task data is input into the large model object under test, and test execution link data is collected; Based on the rule boundary data, the boundary state data, and the test execution link data, identify out-of-bounds content output by the tested large model object that exceeds the usage boundary represented by the rule boundary data, determine the knowledge source and triggering path corresponding to the out-of-bounds content, and generate out-of-bounds attribution data; Red team test result data is generated based on the out-of-bounds attribution data, and the red team test result data is used to characterize the risk of out-of-bounds output of knowledge content of the tested large model object.

Images