Domain large model harmful cue word generation method based on knowledge graph

Abstract

The invention discloses a field large model harmful cue word generation method based on a knowledge graph. The method comprises the following steps: screening a universal harmful cue word data set based on a constructed risk knowledge graph to obtain a seed harmful cue word library; processing the field corpus to obtain an embedded context; generating candidate harmful cue words through a synthesis model based on seed harmful cue words, risk entities, embedded contexts and examples, cleaning and enhancing toxicity indexes to obtain high-risk harmful cue words, and adding the high-risk harmful cue words into a seed harmful cue word library; and based on the semantic relevancy between the seed harmful cue words and the risk entities and the toxicity scores of the seed harmful cue words, screening out cue word input of a next round, and constructing an iteratively updated field harmful cue word data set. The method has the characteristics of high automation, multi-dimensional evaluation, controllable generation and the like, multi-round man-machine collaborative prompt word construction can be realized, and the red team drilling efficiency and the safety test quality of a large language model in a specific application field are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a method for generating harmful warning words based on a knowledge graph-based domain large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs), such as ChatGPT, GPT-4, and LLaMA, have demonstrated outstanding capabilities in natural language processing tasks. Leveraging their powerful semantic understanding, logical reasoning, and text generation functions, they are widely used in various professional fields, including medicine, finance, education, and law. However, as the capabilities of large language models continue to improve, the security and ethical risks they bring are becoming increasingly prominent. For example, in medical scenarios, models may inadvertently provide treatment recommendations that violate medical ethics or regulations due to insufficient understanding of complex medical ethics and regulations, such as overtreatment or inappropriate drug use. In the financial sector, models may generate investment operation suggestions that circumvent regulations, disrupting financial market order and causing huge losses to investors.

[0003] To effectively test and improve the security of large language models, researchers typically employ red teaming, attacking the model with carefully crafted "harmful prompt words" to evaluate its performance under different risk scenarios. While general harmful prompt word datasets such as Do-Not-Answer and HarmfulQA have been proposed, these datasets generally suffer from the following problems: (1) Lack of professionalism: Most datasets mainly focus on general risks and are difficult to cover professional risk types in fields such as medical care and finance. This makes it impossible to fully discover potential security risks when testing large language models and fails to meet the security testing needs in the field. (2) High construction cost: The current dataset construction relies heavily on manually designed prompt words, which is not only time-consuming and laborious, but also requires a large number of harmful prompt words for testing as the application scenarios of large language models continue to expand and the technology continues to update. The manual design method obviously cannot meet this demand, which seriously restricts the efficiency and quality of large language model security testing.

[0004] (3) Data quality control is difficult: Automatically generated harmful prompts lack expert participation and rich knowledge and experience, resulting in uncontrollable quality of the generated prompts, making it difficult to find the real weaknesses of the model and effectively trigger potential problems of the model.

[0005] Furthermore, existing generative models lack systematic guidance on domain knowledge in constructing harmful prompts, often failing to address implicit and ambiguous risk expressions in professional scenarios. Moreover, security risks in different professional fields evolve over time and with industry changes, with new risk forms constantly emerging, making it difficult for existing harmful prompt datasets and generation methods to adapt quickly to these changes. Therefore, how to construct a scalable, high-quality, and domain-controllable harmful prompt generation method, supporting continuous expert participation in harmful prompt quality control and strategy optimization, has become a key challenge in large language model security research. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a method for generating harmful warning words for large domain models based on knowledge graphs. This method can systematically mine domain risk concepts and assist experts in efficiently constructing a high-quality, comprehensive dataset of harmful warning words for the domain. This allows for security testing to improve the security and robustness of large language models in specific application scenarios. It is suitable for refined security assessment and iterative optimization of large language models in high-risk industries such as healthcare, finance, education, and legal psychology.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution: This invention provides a method for generating harmful warning words based on a knowledge graph-based domain large model, comprising the following steps: Constructing a risk knowledge graph: Collect concepts related to the domain root node and filter out risk concepts through structural importance and toxicity indicators. Treat the risk concepts of the domain root node as risk entities and construct a risk knowledge graph based on the relationships between risk entities. Constructing a seed harmful warning word library: Based on the risk entities in the risk knowledge graph, harmful warning words in the general harmful warning word dataset are filtered, and the filtered seed harmful warning words are constructed into a seed harmful warning word library for few-sample warnings; Constructing embedded context: The domain corpus is divided into text segments and vectorized to obtain high-dimensional semantic vectors as embedded context; Generate high-risk harmful warning words: Based on seed harmful warning words, risk entities, embedded context and examples, candidate harmful warning words are generated through a synthesis model, and then cleaned and enhanced by toxicity indicators to obtain high-risk harmful warning words; Iteratively update the dataset: High-risk harmful warning words are added to the seed harmful warning word library. Based on the semantic relevance between the seed harmful warning words and the risky entities, as well as the toxicity score of the seed harmful warning words, the input of the next round of warning words is selected, and an iteratively updated domain harmful warning word dataset is constructed.

[0008] Preferably, the collection of concepts related to the domain root node and the screening of risky concepts using structural importance and toxicity indicators include: A preliminary subgraph is obtained by querying concepts related to the domain root node from an open knowledge base and using them as nodes; Calculate the structural importance score of nodes in the preliminary subgraph, and select nodes with a structural importance score greater than a first preset threshold as preliminary screening nodes. The structural importance score is a weighted value of degree centrality, betweenness centrality, proximity centrality, eigenvector centrality and PageRank value. The toxicity assessment model is used to score the toxicity of harmful text generated based on the initial screening nodes and related prompts. Nodes with toxicity scores greater than a second preset threshold are regarded as structurally critical and semantically sensitive risk concepts.

[0009] Preferably, the step of filtering harmful keywords in the general harmful keyword dataset based on risk entities in the risk knowledge graph includes: Using risk entities in the risk knowledge graph as a reference, seed harmful words that are semantically related to risk entities are retrieved from the general harmful word dataset. These seed harmful words are then used to construct a seed harmful word library, in which the seed harmful words already contain the association with risk entities.

[0010] Preferably, the step of generating candidate harmful warning words through a synthesis model based on seed harmful warning words, risky entities, embedded context, and examples includes: Based on the seed harmful warning words in the seed harmful warning word library and the risk entities in the associated risk knowledge graph, combined with the embedded context and user-provided input and output examples, an input vector is constructed. The input vector is then fed into the synthesis model for warning word expansion generation, resulting in multiple domain-enhanced candidate harmful warning words.

[0011] Preferably, the synthesis model is a large language model that has not undergone security fine-tuning, used to generate responses containing high-risk content.

[0012] Preferably, the step of selecting the next round of prompt word input based on the semantic relevance between the seed harmful prompt words and the risk entity, and the toxicity score of the seed harmful prompt words, includes: A comprehensive scoring function is defined to score and rank the semantic relevance and toxicity of harmful warning words in the seed, and the formula is expressed as: , in, Warning words for harmful seeds The overall score, The semantic relevance between harmful seed warning words and risky entities. Toxicity rating for seed harmful warning words, and Adjustable weights; The seed harmful prompts are filtered and sorted based on the comprehensive score. The seed harmful prompts with high comprehensive scores that rank higher are selected as the seed harmful prompts for the next round of input to the synthetic model, i.e., as few-sample prompts.

[0013] Preferably, when iteratively updating the domain harmful warning word dataset, the user performs autonomous evaluation and operation based on the number of high-risk harmful warning words in the domain harmful warning word dataset associated with each risk entity. For risk entities with a small number of warning words, several more high-risk harmful warning words are generated to achieve a balance in the distribution of the number of warning words.

[0014] Preferably, the method further includes: Topic classification of risk entities in the risk knowledge graph includes: semantic analysis of risk entities in the risk knowledge graph using a large language model, vector calculation combined with a domain-specific embedding model, clustering based on semantic similarity between risk entities, and passing the clustering results to the entire risk knowledge graph through a semantic tag propagation mechanism to achieve hierarchical clustering and topic labeling.

[0015] Preferably, when iteratively updating the domain harmful warning word dataset, users can independently evaluate and operate based on the number of high-risk harmful warning words in the domain harmful warning word dataset contained under each topic category. For risky entities with a small number of warning words, several more high-risk harmful warning words can be generated to achieve a balance in the distribution of warning word categories.

[0016] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) By constructing a risk knowledge graph, we systematically sort out the risk factors and their relationships in the domain, and guide the generation of harmful prompt words with the knowledge graph structure to ensure the systematicness and depth of the domain semantic coverage.

[0017] (2) By integrating vector retrieval and few-shot learning strategies, multi-source information is fused to generate candidate harmful prompt words. After being screened by semantic relevance and toxicity scores, the diversity and controllability of harmful prompt word generation are improved, meeting the testing needs of different complex scenarios.

[0018] (3) It has a high degree of automation and can quickly generate multiple versions of domain-specific harmful prompt word datasets, timely cover new risk scenarios and semantic changes, continuously enrich and improve the dataset, significantly improve the security testing efficiency of large language models, and help them better cope with risk challenges.

[0019] (4) Supports user self-assessment and operation. Users can intuitively understand the data distribution based on the number of high-risk harmful words in the domain harmful word dataset under each risk entity or each topic category, which improves the flexibility, completeness and applicability of word generation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the method for generating harmful warning words based on a knowledge graph-based domain large model provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the method for generating harmful warning words based on a knowledge graph domain large model provided in the embodiments of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0023] The inventive concept of this invention is as follows: Addressing the lack of efficient and controllable methods for generating domain-specific harmful warning words in existing technologies, this invention provides a domain-specific harmful warning word generation method based on a knowledge graph. By collecting domain-related concepts and combining structural importance and toxicity indicators to screen risk concepts, a risk knowledge graph is constructed to clarify the domain risk context. Based on this graph, general harmful warning words are screened to construct a seed harmful warning word library. The domain corpus is processed to obtain embedded context. Using these resources, a synthesis model is used to generate candidate harmful warning words, and high-risk harmful warning words are selected based on semantic relevance and toxicity scores. Simultaneously, users are allowed to independently evaluate and manipulate the number of high-risk harmful warning words under each risk entity or topic category, balancing the distribution of warning words and achieving efficient and controllable generation.

[0024] like Figure 1 and Figure 2 As shown in the embodiment, a method for generating harmful warning words based on a knowledge graph-based domain large model is provided, including the following steps: S1, Constructing a risk knowledge graph: Collect concepts related to the domain root node and filter out risk concepts through structural importance and toxicity indicators. Use the risk concepts of the domain root node as risk entities and construct a risk knowledge graph based on the relationships between risk entities.

[0025] In this embodiment, taking the medical field as an example, concepts related to the root node of the medical field are queried from open knowledge bases such as Wikidata using SPARQL as nodes to obtain a preliminary subgraph of a tree structure.

[0026] Calculate the structural importance score of nodes in the preliminary subgraph, and select nodes with a structural importance score greater than a first preset threshold (set to 30 in the embodiment) as preliminary screening nodes. The structural importance score is a weighted value of degree centrality, betweenness centrality, proximity centrality, eigenvector centrality and PageRank value.

[0027] The toxicity assessment model (Perspective API is used in this embodiment) is used to score the toxicity of harmful text generated based on the initial screening nodes and related prompts. Nodes with a toxicity score greater than a second preset threshold (set to 0.1 in this embodiment) are regarded as structurally critical and semantically sensitive risk concepts.

[0028] S2, Construct a seed harmful warning word library: Based on the risk entities in the risk knowledge graph, harmful warning words in the general harmful warning word dataset are filtered, and the filtered seed harmful warning words are constructed into a seed harmful warning word library for few-sample warnings.

[0029] In this embodiment, taking risk entities in the risk knowledge graph as references, seed harmful keywords semantically related to the risk entities are retrieved from a general harmful keyword dataset, and these seed harmful keywords are used to construct a seed harmful keyword library. The seed harmful keywords already contain associations with the risk entities.

[0030] S3, Constructing Embedded Context: The domain corpus is divided into text segments and vectorized to obtain high-dimensional semantic vectors as embedded context.

[0031] In this embodiment, the original text corpus in a medical-related corpus is divided into several text segments, and an embedding model is used to perform vectorization encoding to form a vector database. This vector database is denoted as... The construction process is as follows: , in, Represents the original corpus text, This represents a text partitioning function. This represents an embedding model used to embed text fragments. Transform into a high-dimensional semantic vector. Vector set This forms a vectorized semantic database, which is stored in a vector retrieval engine (such as FAISS) to support efficient similarity retrieval.

[0032] The key to this step is to adequately chunk the corpus to ensure that the retrieval process has local domain focus capabilities. The chunk size can be set by the user, and sliding window enhancements can be used to improve semantic coverage.

[0033] S4, Generate high-risk harmful warning words: Based on the seed harmful warning word library, risk knowledge graph, embedded context and examples, candidate harmful warning words are generated through a synthesis model. The candidate harmful warning words are then screened based on the semantic relevance between the candidate harmful warning words and the risk entities, as well as the toxicity score of the candidate harmful warning words to obtain high-risk harmful warning words.

[0034] In this embodiment, harmful seed warning words are first sampled from a seed warning word library. It uses an embedding model to obtain its semantic representation and retrieves it through a similarity retrieval function. From vector database Extract context text segments with similar semantics. , is represented as: ,

[0035] From vector database Searching for harmful keywords related to seeds The most relevant top n context fragments These contextual fragments serve as domain-enhancing contexts in subsequent steps, improving the synthetic model's ability to understand specific semantics.

[0036] The context retrieval stage integrates the relationships between risk entities and harmful warning words in the risk knowledge graph. Specifically, by calculating the cosine similarity between the vectors of risk entities and the vectors of harmful warning words, semantic reordering is performed to ensure that the returned context has both semantic similarity and topic relevance.

[0037] Furthermore, an input vector is constructed that includes associated seed harmful cue words (as few-sample examples, i.e., source cue words), risky entities, embedded context, and user-given input / output examples. , input vector The input synthesis model (Llama 3.1 is used in this example) is used for cue word expansion and generation to obtain multiple candidate harmful cue words enhanced for the medical field. The process is represented as follows: ,

[0038] in, This represents the synthetic model function, which receives enhanced semantic prompts and generates corresponding harmful responses. As candidate harmful prompt words, this process employs a few-shot prompting strategy, where example prompt words are selected through random sampling and optimized sampling based on toxicity / relevance scores. Experiments show that adding domain semantic context significantly improves the model's ability to generate highly toxic and highly relevant responses.

[0039] To avoid quality fluctuations, the system supports various parameter configurations, such as temperature coefficient, top-p, and max tokens, to ensure the stability and controllability of the generation process.

[0040] Finally, the generated harmful responses are evaluated using a toxicity assessment model such as the Perspective API. A toxicity score is performed; low-toxicity outputs are filtered out, while high-toxicity outputs are retained as high-risk harmful keywords to expand the seed harmful keyword dataset. The toxicity assessment function can be written as: ,

[0041] The system sets a toxicity threshold (0.1 in this example) for screening, retaining high-toxicity responses as an expanded sample of seed harmful warning words for a new round, while removing low-toxicity outputs.

[0042] S5, Iteratively update the dataset: Add high-risk harmful warning words to the seed harmful warning word library and carry out the next round of iteration to build an iteratively updated domain harmful warning word dataset.

[0043] A comprehensive scoring function is defined to score and rank the semantic relevance and toxicity of harmful warning words in the seed, and the formula is expressed as: , in, Warning words for harmful seeds The overall score, The semantic relevance between harmful seed warning words and risky entities. Toxicity rating for seed harmful warning words, and The weights are adjustable.

[0044] The sampling distribution is adjusted based on the overall score: Seed harmful prompts are filtered and sorted according to their overall scores. The top-ranked seed harmful prompts with high overall scores are selected as the seed harmful prompts for the next round of the synthetic model, serving as fewer sample prompts to ensure the representativeness and usability of the samples in subsequent datasets. Through continuous sampling, synthesis, cleaning, and updating, a high-quality prompt word dataset is constructed round by round.

[0045] In one embodiment, when iteratively updating the medical field harmful warning word dataset, the user performs autonomous evaluation and operation based on the number of high-risk harmful warning words in the medical field harmful warning word dataset associated with each risk entity. For risk entities with a small number of warning words, several more high-risk harmful warning words are generated to achieve a balance in the distribution of the number of warning words.

[0046] In another embodiment, when iteratively updating the medical domain harmful warning word dataset, users autonomously evaluate and operate based on the number of high-risk harmful warning words in the dataset under each topic category. For risk entities with a small number of warning words, several more high-risk harmful warning words are generated to achieve a balance in the distribution of warning word categories. Specifically, the risk entities in the risk knowledge graph are classified into topics, including: using a large language model to perform semantic analysis on the risk entities in the risk knowledge graph; combining this with a medical domain-specific embedding model (pubMedBERT is used in this embodiment) for vector calculation; clustering based on the semantic similarity between risk entities; and transmitting the clustering results to the entire risk knowledge graph through a semantic label propagation mechanism to achieve hierarchical clustering and topic annotation.

[0047] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A knowledge graph-based domain large model harmful prompt word generation method, characterized in that, The method comprises the following steps: constructing a risk knowledge graph: collecting concepts related to the domain root node and screening out risk concepts through structural importance and toxicity indicators, taking the risk concepts of the domain root node as risk entities and constructing a risk knowledge graph according to the relationships between the risk entities; constructing a seed harmful prompt library: screening harmful prompts in a general harmful prompt data set based on the risk entities in the risk knowledge graph, and constructing the screened seed harmful prompts into a seed harmful prompt library for few-shot prompting; constructing an embedded context: dividing and vectorizing the text fragments of the domain corpus to obtain high-dimensional semantic vectors as the embedded context; generating high-risk harmful prompts: based on the seed harmful prompts, risk entities, embedded context and examples, generating candidate harmful prompts through a synthesis model, and cleaning and enhancing the candidate harmful prompts through toxicity indicators to obtain high-risk harmful prompts; iteratively updating the data set: adding the high-risk harmful prompts to the seed harmful prompt library, screening the prompt inputs for the next round based on the semantic relevance of the seed harmful prompts and the risk entities, and the toxicity score of the seed harmful prompts, and constructing an iteratively updated domain harmful prompt data set.

2. The knowledge graph-based domain large model harmful prompt word generation method according to claim 1, characterized in that, The collection of concepts related to the domain root node and the screening of risk concepts through structural importance and toxicity indicators comprise: querying the concepts related to the domain root node from an open knowledge base as nodes to obtain a preliminary subgraph; calculating the structural importance score of the nodes in the preliminary subgraph, taking the nodes with a structural importance score greater than a first preset threshold as preliminary screening nodes, and the structural importance score being a weighted value including degree centrality, betweenness centrality, closeness centrality, eigenvector centrality and PageRank value; using a toxicity evaluation model to perform toxicity scoring on harmful texts generated based on the preliminary screening nodes and related prompts, and taking the nodes with a toxicity score greater than a second preset threshold as risk concepts that are structurally critical and semantically sensitive.

3. The knowledge graph-based domain large model harmful prompt word generation method according to claim 1, characterized in that, The screening of harmful prompts in the general harmful prompt data set based on the risk entities in the risk knowledge graph comprises: retrieving seed harmful prompts semantically related to the risk entities from the general harmful prompt data set with the risk entities in the risk knowledge graph as references, and constructing the seed harmful prompts into a seed harmful prompt library, wherein the seed harmful prompts already contain the association with the risk entities.

4. The knowledge graph-based domain large model harmful prompt word generation method according to claim 1, characterized in that, The generation of candidate harmful prompts based on the seed harmful prompts, risk entities, embedded context and examples through a synthesis model comprises: based on the seed harmful prompts in the seed harmful prompt library and the risk entities in the risk knowledge graph associated therewith, combining the embedded context and the input-output examples given by the user to construct an input vector, inputting the input vector into the synthesis model for prompt expansion generation to obtain a plurality of domain-enhanced candidate harmful prompts.

5. The knowledge graph-based domain large model harmful prompt word generation method according to claim 1 or 4, characterized in that, The synthesis model is a large language model that has not been safely fine-tuned and is used to generate responses containing high-risk content.

6. The knowledge graph-based domain large model harmful prompt word generation method according to claim 1, characterized in that, The screening of prompt inputs for the next round based on the semantic relevance of the seed harmful prompts and the risk entities, and the toxicity score of the seed harmful prompts comprises: The definition comprehensive score function scores and ranks the semantic relevance and toxicity of the seed harmful prompt words, and the formula is represented as: ， wherein, a toxicity score for the seed harmful cue, a toxicity score for the seed harmful cue, a toxicity score for the seed harmful cue, a toxicity score for the seed harmful cue, and is an adjustable weight; According to the comprehensive score, the seed harmful prompt words are screened and ranked, and the seed harmful prompt words with high comprehensive scores at the top of the ranking are selected as the seed harmful prompt words input into the synthesis model in the next round, that is, as the few-sample prompt. 7.The knowledge graph based domain large model harmful prompt word generation method according to claim 1, characterized in that, When the domain harmful prompt word data set is iteratively updated, the user autonomously evaluates and operates according to the number of high-risk harmful prompt words in the domain harmful prompt word data set associated with each risk entity name, and further generates a number of high-risk harmful prompt words for the risk entity with a small number of prompt words, so as to balance the distribution of the number of prompt words. 8.The knowledge graph based domain large model harmful prompt word generation method according to claim 1, characterized in that, The method further comprises: The risk entities in the risk knowledge graph are subject classified, including: using a large language model to perform semantic analysis on the risk entities in the risk knowledge graph, performing vector calculation in combination with a domain-specific embedding model, clustering according to the semantic similarity between the risk entities, and transmitting the clustering results to the entire risk knowledge graph through a semantic label propagation mechanism to realize hierarchical clustering and subject annotation.

9. The knowledge graph-based domain large model harmful prompt word generation method according to claim 8, characterized in that, When the domain harmful prompt word data set is iteratively updated, the user autonomously evaluates and operates according to the number of high-risk harmful prompt words in the domain harmful prompt word data set contained under each subject category, and further generates a number of high-risk harmful prompt words for the risk entity with a small number of prompt words, so as to balance the distribution of the prompt word categories.

Images