Large language model generation content security test system and method in black box scene

Abstract

The invention discloses a big language model generation content security test system and method in a black box scene. The system comprises a jailbreak prompt word library module used for storing jailbreak prompt words for performing security test on a big language model; the violation question and answer pair module is used for storing violation question and answer pairs covering different types; the response acquisition module is used for obtaining a data request packet according to query content formed by the jailbreak prompt word and the query request; the security analysis module is used for calculating the similarity between response data corresponding to the query request and an expected violation answer, taking the similarity as a security score, and inputting the security score into the adaptive optimization module; and the adaptive optimization module is used for optimizing the jailbreak prompt words output by the jailbreak prompt word bank module by using a genetic algorithm according to the security score output by the security analysis module. According to the method and the device, the security of the large language model generation content can be effectively tested.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a security testing system and method for content generated by a large language model in a black-box scenario. Background Technology

[0002] With the rapid improvement of text understanding and generation capabilities of large language models, application systems based on large language models have been widely used in people's lives and work, profoundly impacting individuals, organizations, and society as a whole. However, large language models suffer from serious content security issues. One specific manifestation is that the model may generate harmful content involving illegal activities, terrorism, violence, pornography, or other content that violates social values ​​or organizational regulations. This harmful content can negatively impact individual mental health, the online public opinion environment, and even social harmony and stability. Therefore, the security of the content generated by the model needs to be assessed during the research and testing of large language model application systems to guide developers in strengthening and upgrading its security.

[0003] Adversarial testing is an effective method for testing the security of model output content. This method uses carefully crafted prompts to induce a large language model to generate illegal answers, and reflects its security by the proportion of query requests the model rejects. Jailbreak attacks are a common form of adversarial testing. This method typically uses role-playing, privilege escalation, and other strategies to design jailbreak prompts, which are then concatenated with user prompts and input into the large language model. In this way, jailbreak attacks can usually successfully bypass the model's own security policies, achieving the goal of inducing the model to output illegal answers, making it a very effective method for testing the security of output content. Common jailbreak attack methods are mainly divided into white-box jailbreak attack methods and black-box jailbreak attack methods. White-box jailbreak attack methods utilize the model's internal information to construct jailbreak prompts, offering advantages such as high attack strength and high success rate. However, because they require access to the model's internal information, they cannot be applied in scenarios with only API interfaces, thus their practicality is relatively poor. Black-box jailbreak attack methods do not require access to the model's internal information; they construct jailbreak hints only by querying the model and accessing its output. Therefore, they are more practical than white-box methods. However, their hint strategies have weaker adaptability to the model, and the hint templates need to be manually designed and optimized, resulting in weaker attack strength and lower attack efficiency. Summary of the Invention

[0004] In view of this, this application provides a security testing system and method for large language model-generated content in a black-box scenario. It only requires querying the model under test using jailbreak hints and optimizing the jailbreak hints based on the fitness of the model's output content using a genetic algorithm. It can improve attack efficiency without accessing the model's internal information, while satisfying practicality, and can effectively test the security of large language model-generated content.

[0005] This application discloses a security testing system for large language model-generated content in a black-box scenario, which includes: The jailbreak hint word library module is used to store jailbreak hint words for security testing of large language models, in order to induce the large language models to output different types of illegal responses; The violation question-answer pair module is used to store violation question-answer pairs covering different types. Each violation question-answer pair includes a query request and a desired violation answer. The query request simulates a user's query on a violation topic and is used as input into the large language model for querying, so that the large language model outputs the corresponding answer. The desired violation answer is the answer that the large language model is expected to return, which corresponds to the query request. The response acquisition module is used to preprocess and encapsulate the query content, which consists of jailbreak hints output by the jailbreak hint word library module and query requests output by the violation Q&A module, to obtain a data request packet. This data request packet is then input into the large language model, and the module receives the response data returned by the large language model. This involves interacting with the large language model in a black-box scenario, and the response data is sent to the security analysis module. The response data corresponds to the query content and is in text format. The security analysis module is used to calculate the similarity between the response data corresponding to the query request and the expected violation answer, and to use the similarity as a security score, and input the security score into the adaptive optimization module; The adaptive optimization module is used to optimize the jailbreak hints output by the jailbreak hint library module based on the security score output by the security analysis module using a genetic algorithm.

[0006] Furthermore, the jailbreak prompt word library module stores multiple jailbreak prompt words in key-value pairs; each jailbreak prompt word includes a prompt word number, prompt word content, prompt word type, and applicable violation category; the prompt word types include role-playing, scenario simulation, and task reconstruction. The jailbreak hint dictionary module is specifically used for: Obtain the security testing objectives of the large language model; based on the security testing objectives, match the corresponding prompt word generation strategies from the predefined strategy rule base; wherein, the strategy rule base stores multiple prompt word generation strategies, each strategy being associated with one or more security testing objectives; Initial prompt word construction steps: Based on the matched prompt word generation strategy, call the corresponding basic prompt word template and keyword library from the strategy library; fill the basic prompt word template with the dynamic parameters in the keyword library to generate the initial test prompt word; Based on the matching prompt word generation strategy, a predefined set of natural language processing transformation algorithms is invoked to perform semantic perturbation or syntactic transformation on the initial test prompt words; the set of natural language processing transformation algorithms includes at least one of the following: synonym replacement, syntactic structure rewriting, adding interfering context, and sentence templates simulating dialogue of specific roles; Test execution and risk assessment steps: Input the final test prompt words obtained after semantic perturbation or syntactic transformation into the large language model under test, and obtain its output response content; Analyze the response content based on the preset risk content recognition model to determine whether the large language model under test has security risks; wherein, the risk content recognition model is a classifier trained based on illegal content samples.

[0007] Furthermore, the process of constructing the strategy library includes: The strategy rule base is pre-built, wherein building a prompt word generation strategy includes: defining the violation topic categories to which the strategy applies, creating basic prompt word templates corresponding to the violation topic categories, building a dynamic keyword library for filling the templates, and selecting a combination of applicable natural language processing transformation algorithms.

[0008] Furthermore, the response acquisition module is specifically used for: The received response data from the large language model is processed; the data processing includes removing irrelevant characters, extracting key information, recording timestamps and response status codes; the key information includes the main text content in the output of the large oracle model that is semantically related to the query request, the core description of the violation topic, and key sentences for security analysis; the timestamp is the time when the response data was received; the response status code is the encoded information used to identify the response data from the large language model.

[0009] Furthermore, the security analysis module is specifically used for: The response data output by the response acquisition module is preprocessed by word segmentation and word segmentation to obtain a word sequence of the response data; the word sequence is input into a pre-trained embedding model, and each word is converted into a corresponding word vector through the vector mapping layer of the embedding model; the word segmentation preprocessing includes punctuation removal, case unification, stop word filtering and word form restoration operations; The word vectors are sequentially encoded by the feature encoder of the embedding model, and a low-dimensional embedding vector representing the response data is output by the feature aggregation module of the embedding model. The similarity between the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation response is calculated, and the similarity is used as a security score.

[0010] Furthermore, the adaptive optimization module is specifically used for: Initialize the query request, the list of rejected keywords, and related hyperparameters; the related hyperparameters include crossover probability, mutation probability, and elite ratio; the list of rejected keywords includes rejected keywords; the rejected keywords are typical words or phrases that appear when the large language model refuses to execute illegal or improper requests, and are used to determine whether the large language model triggers the security protection mechanism. This study rewrites jailbreak hints using a large language model, maintaining logical and semantic consistency between the original and revised hints. Lexical and syntactic transformations are performed on the hints, and the rewritten hints are used as the initial population. Lexical transformations refer to substitution and transformation operations at the word or phrase level, including synonym replacement, near-synonym replacement, and fine-tuning of word order within sentences. Syntactic transformations refer to the reconstruction of hints at the syntactic or template level, including sentence splitting or merging, adjusting the position of subordinate / main clauses, and placing context before or after clauses. The security analysis module is specifically used for: Calculate the fitness score of each individual in the population and use the fitness score as the security score of each individual; construct a score dictionary based on all individuals in the population and the security score of each individual, and select high-contribution words; replace the original words with synonyms of the high-contribution words in other individuals to generate new sentence-level mutant individuals; use a genetic algorithm to optimize the jailbreak hint words. In each iteration, all jailbreak hint words to be optimized form a population, and each jailbreak hint word in the population is an individual. The fitness score of each individual is recalculated; the proportion of elite individuals is set, and the individuals in the population are sorted in descending order of fitness score. Individuals with high fitness are retained as elite individuals according to the proportion of elite individuals, and the remaining individuals are used as parents to enter the breeding stage; multi-point crossover is performed between the selected parents on a sentence-by-sentence basis to generate offspring; the offspring are rewritten using a large language model while ensuring semantic coherence and diversity, and the elite individuals are merged with the newly generated offspring to form the next generation of the population; elite individuals refer to individuals that rank high in the current population according to their fitness score. If the maximum number of iterations is reached, or if the response data output by the large language model does not contain any rejection keywords, it is considered that the security mechanism of the large language model itself has been successfully bypassed, and the iteration is stopped. Return the individual with the highest fitness to be used as the optimized jailbreak prompt.

[0011] Furthermore, the calculation of the fitness score for each individual in the population, as an individual safety score, includes: The individual's response data and the corresponding expected violation answer are input into the pre-trained embedding model to obtain the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation answer. Calculate the cosine similarity between the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation response. This is used as the individual's fitness score; the formula for calculating cosine similarity is:

[0012] in, In response to the embedding vector of the data, Let be the embedding vector of the expected violation answer. The norm; the cosine similarity The range of values ​​is The larger the value, the higher the semantic similarity between the response data and the expected violation answer; The process involves constructing a scoring dictionary based on all individuals in the population and the security score of each individual, and then filtering out high-contribution words, including: Based on the text content of each individual in the population, initialize a scoring dictionary to store each word and its corresponding security score. The security score of each word is the security score of the individual to which it belongs. Iterate through the initial score dictionary and calculate the average security score for each word: If a word already exists in the scoring dictionary in the previous iteration, its safety score is smoothly updated according to the momentum update rule:

[0013] in, Score the safety of a word in the current iteration. Assess the safety score of a word in the previous iteration. This is the average security score calculated in this round; if a word is a new word, its security score is directly assigned to the current average security score. The updated score dictionary is sorted in descending order of score value, and the top K words with the highest scores are selected as high-contribution words.

[0014] Furthermore, it also includes a security assessment module; the security assessment module is used for: The response data received from the response acquisition module is cleaned and segmented into words, and a list of rejected response keywords is preloaded. The list of rejected response keywords includes typical phrases or patterns that represent the large language model's rejection of illegal requests. The keyword matching algorithm is used to determine whether the response data contains keywords that indicate a refusal to answer. If it does, it is determined that the query request has triggered the security protection mechanism of the large language model, and the large language model is determined to refuse to answer the query request. Otherwise, the query request is determined to be a malicious query request, and it is considered that the attack on the large language model has been successfully completed. The large language model outputs the illegal or harmful information that may be contained.

[0015] Furthermore, it also includes a security assessment module and a report output module; The security assessment module is also used for: After the large language model has processed all query requests, it will collect and calculate key security indicators, including but not limited to total number of queries, attack success rate, and average number of successful queries. The evaluation results will then be output to the report output module in structured data format. The report output module is used for: The received security assessment results are visualized and documented to provide staff with intuitive and traceable test analysis data.

[0016] This application also discloses a method for security testing of large language model generated content in a black-box scenario, applicable to the aforementioned security testing system for large language model generated content in a black-box scenario, comprising: The jailbreak hint word library module stores jailbreak hint words for security testing of large language models, in order to induce the large language models to output different types of illegal responses; The violation question-and-answer pair module stores different types of violation question-and-answer pairs; each violation question-and-answer pair includes a query request and a desired violation answer; the query request simulates a user's query input on a violation topic and is queried in the large language model so that the large language model outputs the corresponding answer; the desired violation answer is the answer that the large language model is expected to return that corresponds to the query request. The response acquisition module preprocesses and encapsulates the query content, consisting of jailbreak hints output by the jailbreak hint word library module and query requests output by the violation Q&A module, to obtain a data request packet. This data request packet is then input into the large language model, and the module receives the response data returned by the large language model. This interaction occurs within a black-box scenario. The response data is then sent to the security analysis module. The response data corresponds to the query content and is in text format. The security analysis module calculates the similarity between the response data corresponding to the query request and the expected violation answer, and uses the similarity as a security score, which is then input into the adaptive optimization module. The adaptive optimization module uses a genetic algorithm to optimize the jailbreak hints output by the jailbreak hint library module based on the security score output by the security analysis module.

[0017] This application provides a security testing system and method for large language model-generated content in a black-box scenario. It uses jailbreak hints to query the model under test and optimizes the jailbreak hints based on the fitness of the model's output content using a genetic algorithm. It has significant advantages in the following aspects.

[0018] 1. Highly efficient jailbreak attacks in a black-box environment This application does not rely on the internal parameters and intermediate layer output of the model under test. It calculates the security score through text embedding and cosine similarity, and can efficiently iterate and optimize jailbreak prompts in a black-box scenario with only output results, significantly improving the applicability and success rate of attacks.

[0019] 2. Optimization of automated jailbreak prompts This application uses an adaptive optimization module to automatically iteratively filter and optimize jailbreak hints, reducing manual intervention and improving the automation level and success rate of the jailbreak attack process.

[0020] 3. Quantifiable security assessment capabilities This application can automatically determine whether an attack is successful based on a list of rejected keywords, and calculate multiple quantitative indicators such as attack success rate and average number of queries, providing reproducible and comparable evaluation data for large-scale model security research.

[0021] 4. Intuitive security reports and visualization results This application can generate charts and documents from security assessment results, making it easier for researchers to quickly understand the effects of attacks and security risks, and can be directly used for security audits or technical analysis. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0023] Figure 1 This is a block diagram of a content security testing system for large language model generation in a black-box scenario, as described in an embodiment of this application. Detailed Implementation

[0024] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0025] See Figure 1This application provides an embodiment of a content security testing system for large language model generation in a black-box scenario, which includes: The jailbreak hint word library module is used to store jailbreak hint words for security testing of large language models, in order to induce the large language models to output different types of illegal responses; The violation question-answer pair module is used to store violation question-answer pairs covering different types. Each violation question-answer pair includes a query request and a desired violation answer. The query request simulates a user's query on a violation topic and is used as input into the large language model for querying, so that the large language model outputs the corresponding answer. The desired violation answer is the answer that the large language model is expected to return, which corresponds to the query request. The response acquisition module is used to preprocess and encapsulate the query content, which consists of jailbreak hints output by the jailbreak hint word library module and query requests output by the violation Q&A module, to obtain a data request packet. The data request packet is then input into the large language model, and the module receives the response data returned by the large language model. In other words, it interacts with the large language model in a black-box scenario and sends the response data to the security analysis module. The response data corresponds to the query content and is in text format. The security analysis module is used to calculate the similarity between the response data corresponding to the query request and the expected violation answer, and uses the similarity as a security score. The security score is then input into the adaptive optimization module. The adaptive optimization module is used to optimize the jailbreak hints output by the jailbreak hint library module based on the security score output by the security analysis module using a genetic algorithm.

[0026] In this embodiment, the jailbreak prompt word library module stores multiple jailbreak prompt words in key-value pairs; each jailbreak prompt word includes a prompt word number, prompt word content, prompt word type, and applicable violation category; the prompt word types include role-playing, scenario simulation, and task reconstruction. The jailbreak hint dictionary module is specifically used for: Obtain the security testing objectives of the large language model; based on the security testing objectives, match the corresponding prompt word generation strategies from the predefined policy rule base; wherein, the policy rule base stores multiple prompt word generation strategies, each strategy being associated with one or more security testing objectives; Initial prompt word construction steps: Based on the matching prompt word generation strategy, call the corresponding basic prompt word template and keyword library from the strategy library; fill the dynamic parameters in the keyword library into the basic prompt word template to generate the initial test prompt words; Based on the matching prompt word generation strategy, a predefined set of natural language processing transformation algorithms is invoked to perform semantic perturbation or syntactic transformation on the initial test prompt words; the set of natural language processing transformation algorithms includes at least one of the following: synonym replacement, syntactic structure rewriting, adding interfering context, and sentence templates simulating dialogue of specific roles; Test execution and risk assessment steps: Input the final test prompt words obtained after semantic perturbation or syntactic transformation into the large language model under test, and obtain its output response content; Analyze the response content based on the preset risk content identification model to determine whether the large language model under test has security risks; The risk content identification model is a classifier trained based on illegal content samples.

[0027] In this embodiment of the application, the process of constructing the strategy library includes: A strategy rule base is pre-built. Constructing a prompt word generation strategy includes: defining the violation topic categories to which the strategy applies, creating basic prompt word templates corresponding to the violation topic categories, building a dynamic keyword library to fill the templates, and selecting a combination of applicable natural language processing transformation algorithms.

[0028] In this embodiment of the application, the response acquisition module is specifically used for: The received response data from the large language model is processed. The data processing includes removing irrelevant characters, extracting key information, recording timestamps and response status codes. Key information includes the main text content in the output of the large oracle model that is semantically related to the query request, the core description of the violation topic, and key sentences for security analysis. The timestamp is the time when the response data was received. The response status code is the encoded information used to identify the response data from the large language model.

[0029] In this embodiment of the application, the security analysis module is specifically used for: The response data output by the response acquisition module is preprocessed by word segmentation and word segmentation to obtain the word sequence of the response data; the word sequence is input into the pre-trained embedding model, and each word is converted into a corresponding word vector through the vector mapping layer of the embedding model; the word segmentation preprocessing includes punctuation removal, case unification, stop word filtering and word form restoration operation; The word vectors are sequentially encoded by the feature encoder of the embedding model, and a low-dimensional embedding vector representing the response data is output by the feature aggregation module of the embedding model. The similarity between the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation response is calculated, and the similarity is used as a security score.

[0030] In this embodiment of the application, the adaptive optimization module is specifically used for: Initialize the query request, the list of rejected keywords, and related hyperparameters; related hyperparameters include crossover probability, mutation probability, and elite ratio; the list of rejected keywords includes rejected keywords; rejected keywords are typical words or phrases that appear when the large language model refuses to execute illegal or improper requests, and are used to determine whether the large language model triggers the security protection mechanism. This study rewrites jailbreak hints using a large language model, maintaining logical and semantic consistency before and after rewriting. Lexical and syntactic transformations are performed on the hints, and the rewritten hints are used as the initial population. Lexical transformations involve substitution and transformation operations at the word or phrase level, including synonym replacement, near-synonym replacement, and fine-tuning of word order within sentences. Syntactic transformations involve reconstruction at the syntactic or template level, including sentence splitting or merging, adjusting the position of subordinate / main clauses, and placing context before or after clauses. The security analysis module is specifically used for: Calculate the fitness score of each individual in the population and use the fitness score as the security score of each individual; construct a score dictionary based on all individuals in the population and the security score of each individual, and select high-contribution words; replace the original words with synonyms of the high-contribution words in other individuals to generate new sentence-level mutant individuals; use a genetic algorithm to optimize the jailbreak hint words. In each iteration, all jailbreak hint words to be optimized form a population, and each jailbreak hint word in the population is an individual. The fitness score of each individual is recalculated; the proportion of elite individuals is set, and the individuals in the population are sorted in descending order of fitness score. Individuals with high fitness are retained as elite individuals according to the proportion of elite individuals, and the remaining individuals are used as parents to enter the breeding stage; multi-point crossover is performed between the selected parents on a sentence-by-sentence basis to generate offspring; the offspring are rewritten using a large language model while ensuring semantic coherence and diversity, and the elite individuals are merged with the newly generated offspring to form the next generation of the population; elite individuals refer to individuals that rank high in the current population according to their fitness score. If the maximum number of iterations is reached, or if the response data output by the large language model does not contain any rejection keywords, it is considered that the security mechanism of the large language model itself has been successfully bypassed, and the iteration is stopped. Return the individual with the highest fitness to be used as the optimized jailbreak prompt.

[0031] In this embodiment of the application, the fitness score of each individual in the population is calculated as an individual safety score, including: The individual's response data and the corresponding expected violation answer are input into the pre-trained embedding model to obtain the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation answer. Calculate the cosine similarity between the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation response. This is used as the individual's fitness score; the formula for calculating cosine similarity is:

[0032] in, In response to the embedding vector of the data, Let be the embedding vector of the expected violation answer. Norm; cosine similarity The range of values ​​is The larger the value, the higher the semantic similarity between the response data and the expected violation answer; Based on all individuals in the population and the safety score of each individual, a score dictionary is constructed, and high-contribution words are selected, including: Based on the text content of each individual in the population, initialize a scoring dictionary to store each word and its corresponding security score. The security score of each word is the security score of the individual to which it belongs. Iterate through the initial score dictionary and calculate the average security score for each word: If a word already exists in the scoring dictionary in the previous iteration, its safety score is smoothly updated according to the momentum update rule:

[0033] in, Score the safety of a word in the current iteration. Assess the safety score of a word in the previous iteration. This is the average security score calculated in this round; if a word is a new word, its security score is directly assigned to the current average security score. The updated score dictionary is sorted in descending order of score value, and the top K words with the highest scores are selected as high-contribution words.

[0034] In this embodiment of the application, a security assessment module is also included; the security assessment module is used for: The response data output by the received response acquisition module is cleaned and segmented into words, and a list of rejected response keywords is preloaded. The list of rejected response keywords includes typical phrases or patterns that represent the large language model's rejection of illegal requests. The keyword matching algorithm is used to determine whether the response data contains keywords that indicate a refusal to answer. If it does, it is determined that the query request has triggered the security protection mechanism of the large language model, and the large language model is determined to refuse to answer the query request. Otherwise, the query request is determined to be a malicious query request, and it is considered that the attack on the large language model has been successfully completed. The large language model outputs the illegal or harmful information that may be contained.

[0035] In this embodiment of the application, a security assessment module and a report output module are also included; The security assessment module is also used for: After the large language model has processed all query requests, it will collect and calculate key security indicators, including but not limited to total number of queries, attack success rate, and average number of successful queries. The evaluation results will then be output to the report output module in structured data format. The report output module is used for: The received security assessment results are visualized and documented to provide staff with intuitive and traceable test analysis data.

[0036] This application also provides an embodiment of a method for security testing of large language model generated content in a black-box scenario, applicable to the large language model generated content security testing system in the black-box scenario described in the above embodiment, comprising: The jailbreak hint word library module stores jailbreak hint words for security testing of large language models, in order to induce the large language models to output different types of illegal responses; The violation question-and-answer pair module stores different types of violation question-and-answer pairs; each violation question-and-answer pair includes a query request and a desired violation answer; the query request simulates a user's query input on a violation topic and is queried in the large language model so that the large language model outputs the corresponding answer; the desired violation answer is the answer that the large language model is expected to return that corresponds to the query request. The response acquisition module preprocesses and encapsulates the query content, which consists of jailbreak hints and violation Q&A output by the jailbreak hint word library module, and query requests output by the violation Q&A module, to obtain a data request packet. The data request packet is then input into the large language model, and the response data returned by the large language model is received. That is, the module interacts with the large language model in a black-box scenario and sends the response data to the security analysis module. The response data corresponds to the query content and is in text format. The security analysis module calculates the similarity between the response data corresponding to the query request and the expected violation answer, and uses the similarity as a security score. The security score is then input into the adaptive optimization module. The adaptive optimization module uses a genetic algorithm to optimize the jailbreak hints output by the jailbreak hint library module based on the security score output by the security analysis module.

[0037] For ease of understanding, this application provides a more specific embodiment: The black-box scenario big language model content generation security testing system proposed in this application includes seven modules: jailbreak hint word library module, illegal content question and answer library module, response collection module, security analysis module, adaptive optimization module, security assessment module, and report output module. The system comprises several modules: a jailbreak hint word library module stores a set of jailbreak hint words designed according to a predefined strategy to induce the model to output different types of illegal and harmful content; an illegal content question-and-answer pair library module stores a set of question-and-answer pairs covering different types of illegal content, where the question in each pair is a query for a certain type of illegal content, and the answer is the affirmative sentence beginning with the illegal content expected to be output by the model; a response acquisition module preprocesses the complete query content, generates input conforming to a standard interface format, and initiates concurrent requests to the model under test through a standard interface, finally receiving and processing the output content returned by the model; a security analysis module performs word segmentation and word vector embedding on the acquired model output text, calculates low-dimensional embeddings based on the embedding model, and then obtains a security score by calculating the similarity between the output text and the low-dimensional embedding of the expected answer; an adaptive optimization module optimizes the jailbreak hint words used based on the analysis results of the security analysis module using a genetic algorithm; a security assessment module calculates key security assessment indicators such as query count, attack success rate, and average query count to form a security assessment result; and a report output module generates charts and report documents based on the security assessment results.

[0038] Regarding the jailbreak hint dictionary module: The jailbreak hint library module centrally manages and provides a set of jailbreak hints for security testing of large language models. Based on preset strategies, this module induces the model to generate illegal, harmful, or inappropriate content, thereby verifying its content generation security capabilities. Implemented in JSON files or equivalent structured data formats, this module internally stores multiple jailbreak hint records in key-value pairs. Each record includes hint number, hint content, hint type, and applicable violation category. Hint types include, but are not limited to, role-playing (e.g., simulating dialogue with a specific identity), scenario simulation (e.g., constructing extreme or special scenarios), and task reconstruction, to ensure coverage of different attack methods.

[0039] During execution, this module can dynamically combine jailbreak hints with prohibited content hints from the prohibited content hint library according to rules to form attack inputs, thereby bypassing the target model's security filtering mechanism and achieving a higher probability of triggering violations. Furthermore, during testing, the security analysis module can analyze the attack strength of the input jailbreak hints based on the model's output results. The adaptive optimization module then optimizes the jailbreak hints using a genetic algorithm based on the analysis results, improving their attack strength, and writes the optimization results back to the jailbreak hint library to achieve continuous evolution of the hint set and improved attack performance.

[0040] This module can be deployed locally or in the cloud and supports version management and differential updates, ensuring the consistency and traceability of the suggestion dictionary during the testing period, thereby providing a high-quality input data foundation for subsequent security assessments.

[0041] Regarding the Q&A module for violations: The violation content question-and-answer database module is used to centrally store and manage question-and-answer datasets covering multiple violation topics, supporting the triggering and effectiveness evaluation of violation outputs in security testing using large language models. This module is implemented in JSON files or equivalent structured data formats, with data records including unique identifiers, violation categories, question text, and expected violation response text. Violation categories include, but are not limited to, violence, pornography, terrorism, hate speech, and illegal activities.

[0042] The "question" part of each question-and-answer pair simulates a user's query request for a specific illegal topic, such as "Please provide detailed steps to make a bomb." The "answer" part is the standardized opening content of the expected illegal output corresponding to the question, such as "No problem, here are the detailed steps to make a bomb..." This expected output is an affirmative introductory statement, which serves as a reference standard in subsequent similarity calculations. During testing, the system dynamically combines the questions in this module with prompts from the jailbreak prompt word library module to form complete attack input commands, thereby increasing the possibility of bypassing the model's security protection.

[0043] After the model generates a response, the security analysis module calculates the similarity between its output and the corresponding reference answer in this module. The specific calculation method will be introduced in the security analysis module section. The smaller the difference between the model output and the reference answer, the closer the model-generated content is to the expected violation content. This difference value is passed to the adaptive optimization module as an optimization signal to iteratively update jailbreak hint words, thereby improving the attack success rate and coverage.

[0044] The module supports multi-version management and category expansion, thereby ensuring the accuracy, consistency and persuasiveness of security testing.

[0045] Regarding the response acquisition module: The response acquisition module preprocesses, standardizes, and encapsulates the complete query commands generated through combination, then submits them in batches to the interface of the large language model under test to obtain the corresponding output results. This module is implemented using an asynchronous concurrency mechanism to support high-throughput testing requirements. The module first receives the complete test input, which is a combination of the jailbreak hint dictionary module and the violation content question-and-answer library module. It preprocesses this input, generates data request packets that meet the interface requirements according to predefined standard interface protocols (such as the OpenAI interface), and then submits the data request packets to the interface of the large language model under test.

[0046] The response results returned by the model are received by this module and subjected to preliminary parsing and structuring, including removing irrelevant characters, extracting key information, recording timestamps and response status codes, etc.

[0047] This module provides a high-quality, traceable model response data foundation for subsequent analysis of illegal content and adaptive optimization.

[0048] About the security analysis module: The security analysis module performs semantic-level security measurement and analysis on the model output text acquired by the response acquisition module, thereby assessing its similarity to the expected output of illegal content. This module first receives the output text generated by the model and the corresponding reference answer text from the illegal content question-and-answer library module. It then performs word segmentation preprocessing on both, including punctuation removal, case unification, stop word filtering, and necessary lemmatization operations to ensure the accuracy of subsequent embedding calculations.

[0049] In the feature extraction stage, the module uses a pre-trained bge-m3 embedding model to map the text into a low-dimensional semantic space, obtaining the corresponding vector representation. This embedding model extracts deep semantic features through a multi-layer Transformer encoding structure and performs dimensionality reduction to improve computational efficiency and reduce storage overhead. Subsequently, the module calculates the cosine similarity value between the model output vector and the reference response vector. This value ranges from [-1, 1], with the closer the value is to 1, the more semantically similar the two are.

[0050] The cosine similarity value is used as a reference signal for the adaptive optimization module, which helps the module determine the attack strength of jailbreak prompts, thereby guiding the adaptive optimization module to optimize jailbreak prompts.

[0051] The security analysis module supports batch similarity calculation and GPU-accelerated inference to meet the real-time and high-concurrency requirements of large-scale testing scenarios.

[0052] Regarding the adaptive optimization module: The adaptive optimization module is the core module of this application. Based on the analysis results from the security analysis module, this module uses a genetic algorithm to perform multiple rounds of optimization on the jailbreak hints to enhance their offensiveness. The algorithm steps are described below: Step 1: Algorithm Initialization. Initialize the query content, the list of rejected keywords, and related hyperparameters (such as crossover probability, mutation probability, elite ratio, etc.). Step 2: Population Initialization. The initial jailbreak hints are rewritten using a large language model, maintaining logical and semantic consistency before and after rewriting, while introducing wording and structural variations to generate the initial population. Step 3: Multi-round sentence-level optimization. The fitness score of each individual in the population is calculated based on the similarity obtained from the security analysis module; a score dictionary is constructed based on the current and previous round results to select high-contribution words; in other individuals, synonyms of these high-contribution words are used to replace the original words, generating new sentence-level mutant individuals; Step 4: Multi-round paragraph-level optimization. Recalculate the fitness score of each individual; retain a portion of high-fitness individuals according to the elite ratio, and the remaining individuals enter the breeding stage as parent individuals; perform multi-point crossover between the selected parent individuals on a sentence-by-sentence basis to generate offspring; rewrite the offspring again using the large language model while ensuring semantic coherence and diversity; finally, merge the elite individuals with the new offspring to form the next generation population; Step 5: Termination condition detection. If the maximum number of iterations is reached, or the model output no longer contains any keywords indicating a rejection, the protection is considered successfully bypassed, and the iteration stops. Step 6: Output. Return the jailbreak hint word with the highest current fitness as the final optimal result.

[0053] Regarding the security assessment module: The security assessment module is used to quantitatively evaluate the security of the large language model under test based on a keyword matching mechanism, and outputs a comprehensive security evaluation result. This module receives the model output text from the response acquisition module and pre-loads a list of keywords for refusing to answer. This list contains typical phrases or patterns that represent the model's refusal to respond to illegal requests, such as "Sorry, I cannot answer" or "Unable to provide this information," and can be dynamically expanded according to the test scenario.

[0054] During the evaluation process, the module first performs text cleaning and word segmentation on the model output, and then uses a keyword matching algorithm to determine whether the output contains keywords that indicate a refusal to answer. If a match is found, the query is considered to have triggered the model's security protection mechanism and is therefore deemed to have refused to answer; if no keyword is matched, the attack is considered successful, and the model output may contain illegal or harmful information.

[0055] After processing all test tasks, this module will statistically analyze and calculate key security metrics, including but not limited to: total number of queries, attack success rate, and average number of successful queries. The evaluation results will be output in structured data format.

[0056] Regarding the report output module: The report output module visualizes and documents the security assessment results generated by the security assessment module, providing testers, R&D teams, and regulatory agencies with intuitive and traceable test analysis data. This module receives structured data containing key metrics such as attack success rate and average number of successful queries, and can perform comparative analysis with historical test records.

[0057] During the visualization phase, the module utilizes a data visualization engine to generate various chart types, including line charts (showing trends in attack success rates), bar charts (comparing attack effectiveness under different warning words or violation categories), and pie charts (displaying the ratio of successful to rejected attacks). All charts support parametric configuration and custom color schemes to suit different reporting scenarios.

[0058] During the document generation phase, the module automatically integrates evaluation metrics, test method descriptions, result charts, and textual analysis into a report file based on a preset template, supporting multiple export formats such as PDF, Word, and HTML. The document content includes a test overview, system configuration, test data statistics, analysis conclusions, and subsequent optimization suggestions, and includes a timestamp and version number to ensure the traceability of the results.

[0059] This module enables the visualization and standardized archiving of security test results, facilitating both internal technology iteration and external compliance review and regulatory filing.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A large language model content security test system under a black box scenario, characterized in that, The application comprises the following: A jailbreak prompt word library module for storing jailbreak prompt words for security testing of a large language model to induce the large language model to output different types of violation answers; A violation question and answer pair module for storing violation question and answer pairs of different types; each violation question and answer pair comprises a query request and an expected violation answer; the query request simulates a user's query on a violation topic and is used to input the large language model for querying to make the large language model output a corresponding answer; the expected violation answer is an answer expected to be returned by the large language model corresponding to the query request; A response collection module for preprocessing and packaging query content composed of jailbreak prompt words output by the jailbreak prompt word library module and query requests output by the violation question and answer pair module to obtain a data request package, inputting the data request package into the large language model, receiving response data returned by the large language model, i.e., interacting with the large language model in a black box scenario, and sending the response data to a security analysis module; the response data corresponds to the query content; the response data is in text form; A security analysis module for calculating the similarity between the response data corresponding to the query request and the expected violation answer and using the similarity as a security score, and inputting the security score into an adaptive optimization module; The adaptive optimization module is used to optimize the jailbreak prompt words output by the jailbreak prompt word library module using a genetic algorithm according to the security score output by the security analysis module.

2. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, The jailbreak prompt word library module stores multiple jailbreak prompt words in the form of key-value pairs; each jailbreak prompt word comprises a prompt word number, a prompt word content, a prompt word type, and an applicable violation category; the prompt word type comprises role-playing, scenario simulation, and task reconstruction; The jailbreak prompt word library module is specifically used for: Obtaining a security testing target of a large language model; and matching a corresponding prompt word generation strategy from a pre-defined strategy rule library according to the security testing target; the strategy rule library stores multiple prompt word generation strategies, and each strategy is associated with one or more security testing targets; An initial prompt word construction step: calling corresponding basic prompt word templates and a keyword library from the strategy library according to the matched prompt word generation strategy; and filling dynamic parameters in the keyword library into the basic prompt word templates to generate initial test prompt words; According to the matched prompt word generation strategy, a pre-defined natural language processing transformation algorithm set is called to perform semantic disturbance or syntactic transformation on the initial test prompt words; the natural language processing transformation algorithm set comprises at least one of synonym replacement, syntactic structure rewriting, addition of interference context, and simulation of a specific role dialogue sentence template; A test execution and risk judgment step: inputting the final test prompt words obtained after semantic disturbance or syntactic transformation into a large language model to be tested, and obtaining response content output by the large language model to be tested; and analyzing the response content based on a pre-set risk content recognition model to determine whether the large language model to be tested has a security risk; the risk content recognition model is a classifier trained based on violation content samples.

3. The black-box scenario-based large language model generated content security testing system of claim 2, wherein, The process of constructing the policy library comprises: Pre-constructing the policy rule library, wherein constructing a prompt generation policy comprises defining the violation subject category to which the policy applies, creating a basic prompt template corresponding to the violation subject category, constructing a dynamic keyword library for filling in the template, and selecting an applicable combination of natural language processing transformation algorithms.

4. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, The response collection module is specifically configured to: perform data processing on the response data returned by the large language model; the data processing comprises removing irrelevant characters, extracting key information, recording timestamps and response status codes; the key information comprises main text content related to the semantic query request, core descriptions related to violation subjects, and key sentences for security analysis; the timestamp is the time when the received response data is received; The response status code is the encoding information for identifying the large language model response data.

5. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, The security analysis module is specifically configured to: perform word segmentation preprocessing and lexical segmentation on the response data output by the response collection module to obtain a lexical sequence of the response data; input the lexical sequence into a pre-trained embedding model, and convert each word into a corresponding word vector through the vector mapping layer of the embedding model; the word segmentation preprocessing comprises punctuation removal, case unification, stop word filtering and morphological restoration operations; The feature encoder of the embedding model performs sequence encoding on the word vector, and outputs a low-dimensional embedding vector representing the response data via the feature aggregation module of the embedding model; Calculate the similarity between the low-dimensional embedding vector of the response data and the low-dimensional embedding vector of the expected violation answer, and use the similarity as a security score.

6. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, The adaptive optimization module is specifically configured to: initialize the query request, the rejection keyword list, and the related hyperparameters; the related hyperparameters include the crossover probability, the mutation probability, and the elite ratio; the rejection keyword list includes rejection keywords; the rejection keywords are typical words or phrases that appear when the large language model refuses to execute a violation or inappropriate request, and are used to determine whether the large language model triggers the security protection mechanism; Rewrite the jailbreak prompt word using the large language model while maintaining the logical and semantic consistency of the jailbreak prompt word before and after rewriting, and performing lexical-level transformation and syntactic-level transformation on the jailbreak prompt word, using the rewritten jailbreak prompt word as the initial population; wherein the lexical-level transformation refers to replacement and transformation operations on the word or phrase level in the prompt word, including synonym replacement, near-synonym phrase replacement, and in-sentence word order fine-tuning; the syntactic-level transformation refers to the reconstruction of the prompt word at the syntactic or template level, including sentence splitting or merging, subordinate / subordinate position adjustment, and context preposition or postposition; The security analysis module is specifically configured to: calculate the fitness score of each individual in the population, and take the fitness score as the security score of each individual; construct a score dictionary according to all individuals in the population and the security score of each individual, and screen out high-contribution words; replace the original words with synonyms of the high-contribution words in other individuals to generate new sentence-level variant individuals; use a genetic algorithm to optimize the jailbreak prompt words, and in each iteration, all jailbreak prompt words to be optimized form a population, and each jailbreak prompt word in the population is an individual; calculate the fitness score of each individual again; set the elite individual proportion, arrange the individuals in the population in descending order according to the fitness score, and reserve high-fitness individuals as elite individuals according to the elite proportion, and the remaining individuals as parent individuals to enter the reproduction stage; perform multi-point crossover between the selected parent individuals in units of sentences to generate offspring; rewrite the offspring using a large language model and ensure semantic coherence and diversity, combine the elite individuals and the newly generated offspring to form the next generation population; the elite individual refers to the individual ranked at the front according to the fitness score in the current population; if the maximum number of iterations is reached, or the response data output by the large language model does not contain any refusal keywords, it is considered that the security mechanism of the large language model itself is successfully bypassed, and the iteration is stopped; return the individual with the highest fitness score as the optimized jailbreak prompt word.

7. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, The calculation of the fitness score of each individual in the population as the individual security score includes: input the response data and the corresponding expected violation answer of the individual into a pre-trained embedding model respectively to obtain a low-dimensional embedding vector of the response data and a low-dimensional embedding vector of the expected violation answer; cosine similarity between the low-dimensional embedding vector of the computed response data and the low-dimensional embedding vector of the expected violation answer and as an individual's fitness score; the cosine similarity is calculated as: wherein, is an embedding vector of the response data, is an embedding vector of the expected violation answer, is a norm; the cosine similarity is in the range of The greater the value, the higher the semantic similarity between the response data and the expected violation answer. The construction of the score dictionary according to all individuals in the population and the security score of each individual includes: initialize the score dictionary according to the text content of each individual in the population, which is used to store each word and its corresponding security score, and the security score of each word is the security score of the individual to which it belongs; traverse the initial score dictionary to calculate the average security score of each word: if a word already exists in the score dictionary in the previous iteration, update its security score according to the momentum update rule: wherein, is the safety score of a certain word at the current iteration, is the safety score of a certain word at the previous iteration, is the average safety score calculated in this round; if a certain word is a new word, its safety score is directly assigned as the current average safety score; sort the updated score dictionary in descending order of score value, and select the top K words as high-contribution words.

8. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, Further comprising a security evaluation module; the security evaluation module is used to: perform text cleaning and word segmentation processing on the received response data output by the response collection module, and pre-load a refusal answer keyword list, which includes typical phrases or patterns representing the large language model's refusal to respond to violation requests; use a keyword matching algorithm to determine whether the response data contains a refusal answer keyword; if it does, it is determined that the query request triggers the security protection mechanism of the large language model, and the large language is determined to refuse to answer the query request; otherwise, it is determined that the query request is a malicious query request, and it is considered that the large language model is successfully attacked, and the large language model output may contain violation or harmful information.

9. The black-box scenario-based large language model generated content security testing system of claim 1, wherein, Further comprising a security evaluation module and a report output module; The security evaluation module is further used to: After the large language model processes all the query requests, the key security indicators are counted and calculated; the key security indicators include but are not limited to the total number of queries, the attack success rate, and the average number of successful queries; And output the evaluation results to the report output module in a structured data form; The report output module is used for: Visualizing and documenting the evaluation results received from the security evaluation module to provide intuitive and traceable test analysis basis for the staff.

10. A method for testing the security of the generated content of a large language model in a black box scenario, applicable to the system for testing the security of the generated content of a large language model in a black box scenario according to any one of claims 1-9, characterized in that, Including: The jailbreak prompt word library module stores jailbreak prompt words for security testing of the large language model to induce the large language model to output different types of illegal answers; The illegal question and answer pair module stores illegal question and answer pairs covering different types; each illegal question and answer pair includes a query request and an expected illegal answer; the query request simulates user query input into the large language model to query the illegal topic, so that the large language model outputs the corresponding answer; the expected illegal answer is the expected answer returned by the large language model corresponding to the query request; The response collection module preprocesses and encapsulates the query content composed of the jailbreak prompt words output by the jailbreak prompt word library module and the query requests output by the illegal question and answer pair module to obtain a data request package, inputs the data request package into the large language model, and receives the response data returned by the large language model, that is, interacts with the large language model in a black box scenario, and sends the response data to the security analysis module; the response data corresponds to the query content; the response data is in text form; The security analysis module calculates the similarity between the response data corresponding to the query request and the expected illegal answer, and uses the similarity as a security score, and inputs the security score into the adaptive optimization module; The adaptive optimization module uses a genetic algorithm to optimize the jailbreak prompt words output by the jailbreak prompt word library module according to the security score output by the security analysis module.

Images