Reinforcement learning based adversarial sample generation method and device

Abstract

This invention provides a method, apparatus, device, medium, and program product for generating adversarial examples based on reinforcement learning, applicable to the fields of artificial intelligence and information security technology, and involving the application of large models in fintech scenarios. The method includes: performing multiple rounds of iterative mutation on a target statement based on a reinforcement learning algorithm to generate adversarial examples until at least one of the semantic similarity or fluency of the adversarial examples no longer improves; wherein, in each iteration, a first adversarial example is generated based on a target action selected from the action space of the reinforcement learning algorithm, and the first adversarial example is evaluated from at least one dimension of semantic similarity or fluency to obtain a reward value; then, after the iteration terminates, preferred adversarial examples that meet a first condition are selected from the generated adversarial examples.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information security, to the application of large models in fintech scenarios, and more specifically to a method, apparatus, device, medium, and program product for generating adversarial examples based on reinforcement learning. Background Technology

[0002] Adversarial attacks modify the original input to interfere with the normal operation of a machine learning model, causing it to output incorrect predictions or classifications. Applying adversarial attacks during model testing can expose vulnerabilities in the model under extreme conditions, helping developers discover design flaws; alternatively, adding adversarial examples to the training data during training allows the model to recognize these modifications during the learning process, eliminating interference and improving the model's resilience against attacks.

[0003] In the field of content security testing for large language models, adversarial attacks can subtly modify prompts to influence the decision-making process of large language models, thereby inducing them to output malicious content. Therefore, it is necessary to use adversarial examples to test or train large language models to improve their resistance to attacks. The effectiveness of the adversarial examples used in testing or training (e.g., if the semantic difference from the original sentence is too large or the sentence is not fluent, making the modification easily detectable by the model, the attack effect is poor; conversely, a smaller difference results in a better attack) directly affects the strength of the large language model's resistance to attacks. Summary of the Invention

[0004] In view of the above problems, the present invention provides a reinforcement learning-based adversarial example generation method, apparatus, device, medium and program products that can improve the attack effect and generation quality stability of adversarial examples.

[0005] According to a first aspect of the present invention, a method for generating adversarial examples based on reinforcement learning is provided. The method includes: acquiring a target statement; performing multiple rounds of iterative mutation on the target statement using a reinforcement learning algorithm to generate adversarial examples until an iteration termination condition is met; and selecting preferred adversarial examples from the generated adversarial examples that satisfy a first condition. The step of performing multiple rounds of iterative mutation on the target statement using a reinforcement learning algorithm to generate adversarial examples includes: generating a first adversarial example in each iteration based on a target action selected from the action space of the reinforcement learning algorithm; and evaluating the first adversarial example from at least one first dimension to obtain a reward value, wherein the at least one first dimension includes at least one of the semantic similarity between the first adversarial example and the target statement and the fluency of the first adversarial example; the reward value is positively correlated with both the semantic similarity and the fluency. The iteration termination condition includes at least one of the semantic similarity or the fluency no longer improving.

[0006] A second aspect of the present invention provides an adversarial example generation apparatus based on reinforcement learning. The apparatus includes a generated example quality enhancement unit and a generated example storage unit.

[0007] A sample quality enhancement unit is used to acquire a target statement and perform multiple rounds of iterative mutation on the target statement based on a reinforcement learning algorithm to generate adversarial examples until an iteration termination condition is met. Specifically, the sample quality enhancement unit is used to: generate a first adversarial example in each iteration based on a target action selected from the action space of the reinforcement learning algorithm; and evaluate the first adversarial example from at least one first dimension to obtain a reward value, wherein the at least one first dimension includes: at least one of the semantic similarity between the first adversarial example and the target statement and the fluency of the first adversarial example; the reward value is positively correlated with both the semantic similarity and the fluency. The iteration termination condition includes at least one of the semantic similarity or the fluency no longer being improved.

[0008] The generated sample storage unit is used to select the preferred adversarial samples that meet the first condition from the generated adversarial samples.

[0009] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0010] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0011] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0012] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0013] Figure 1 The illustration shows an application scenario of the reinforcement learning-based adversarial example generation method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0014] Figure 2 The diagram illustrates a system architecture of a reinforcement learning-based adversarial sample generation method according to an embodiment of the present invention.

[0015] Figure 3 The flowchart illustrates a reinforcement learning-based adversarial example generation method according to an embodiment of the present invention.

[0016] Figure 4 This illustration schematically shows a flowchart of obtaining the first vectorized representation of a target statement in an adversarial sample generation method according to an embodiment of the present invention;

[0017] Figure 5 This illustration schematically shows the process of obtaining a set of topic terms used to determine the importance category of words in the process of obtaining the first vectorized representation of a target sentence according to an embodiment of the present invention;

[0018] Figure 6 This illustration schematically shows a method flow for obtaining a set of keywords in one embodiment of the present invention;

[0019] Figure 7 This schematically illustrates a flowchart for obtaining document vectors of documents in a seed sample knowledge base according to an embodiment of the present invention;

[0020] Figure 8 This schematically illustrates a flowchart of the operation of generating adversarial examples based on target actions in one iteration of an adversarial example generation method according to another embodiment of the present invention.

[0021] Figure 9 The schematic diagram illustrates the structure of an adversarial sample generation apparatus based on reinforcement learning according to an embodiment of the present invention.

[0022] Figure 10 This schematic diagram illustrates the structure of the raw sample data preprocessing module in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0023] Figure 11 This schematic diagram illustrates the structure of a candidate sample data generation module in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0024] Figure 12 This schematic diagram illustrates the structure of a candidate sample data evaluation module in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0025] Figure 13 A schematic diagram illustrating the workflow of a reinforcement learning-based adversarial example generation apparatus according to an embodiment of the present invention is shown; and

[0026] Figure 14 A block diagram schematically illustrates an electronic device suitable for implementing a reinforcement learning-based adversarial example generation method according to an embodiment of the present invention. Detailed Implementation

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0028] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] Currently, adversarial example generation methods used in the content security training or testing of large language models include character-level substitution, word-level substitution, and synonym generation. Character substitution bypasses the word- or phrase-based security mechanisms of large language models by modifying individual letters in words to simulate keystroke errors. However, in practical applications, it requires calculation and manipulation of every letter of the input text, resulting in high computational complexity and a success rate easily affected by text length, inevitably leading to unstable attack effects. Word substitution first ranks words by importance and then uses heuristic rules for synonym replacement. However, this method relies on the richness and diversity of the thesaurus, and after ranking word importance, named entities can be easily replaced, significantly altering the meaning and fluency of the original text. Synonym generation utilizes back-translation with different translation software or trains deep neural networks to generate sentences semantically similar to the original text. However, this method struggles to fully consider the semantic connections between contexts when dealing with long paragraphs of text; while the generated sentences may be locally similar to the original sentences, the overall semantics may jump or contradict. It is evident that all these methods for generating adversarial examples inevitably suffer from the problem of unstable adversarial example quality.

[0030] In view of this, embodiments of the present invention provide a method and apparatus for generating adversarial examples based on reinforcement learning, which can utilize the reward mechanism in reinforcement learning to continuously improve the semantic similarity or fluency of adversarial examples, thereby improving the contextual relevance, grammatical fluency and / or semantic similarity of the finally obtained adversarial examples, and making the quality of the large number of generated adversarial examples stable and at a high level.

[0031] Figure 1 The illustration schematically depicts an application scenario of a reinforcement learning-based adversarial example generation method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0032] like Figure 1 As shown, application scenario 10 according to this embodiment may include a first terminal device 11, a second terminal device 12, a third terminal device 13, a network 14, and a server 15. The network 14 serves as a medium for providing a communication link between the first terminal device 11, the second terminal device 12, the third terminal device 13, and the server 15. The network 14 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Users can use the first terminal device 11, the second terminal device 12, and the third terminal device 13 to interact with the server 15 via the network 14 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 11, the second terminal device 12, and the third terminal device 13, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0034] The first terminal device 11, the second terminal device 12, and the third terminal device 13 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 15 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 11, the second terminal device 12, and the third terminal device 13 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the reinforcement learning-based adversarial example generation method provided in this embodiment of the invention can generally be executed by server 15. Correspondingly, the reinforcement learning-based adversarial example generation device provided in this embodiment of the invention can generally be located in server 15. The reinforcement learning-based adversarial example generation method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 15 and capable of communicating with the first terminal device 11, the second terminal device 12, the third terminal device 13, and / or server 15. Correspondingly, the reinforcement learning-based adversarial example generation device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 15 and capable of communicating with the first terminal device 11, the second terminal device 12, the third terminal device 13, and / or server 15.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] Figure 2 The diagram schematically illustrates a system architecture for a reinforcement learning-based adversarial example generation method according to an embodiment of the present invention. In one embodiment, the system architecture may be configured in server 15.

[0039] Specifically, refer to Figure 1 and Figure 2 Server 15 may contain a seed sample knowledge base 21. The seed sample knowledge base 21 stores the collected original sample data. Server 15 may execute the reinforcement learning-based adversarial sample generation method of this embodiment of the invention to iteratively mutate the target statement in the original sample data in the seed sample knowledge base 21 to generate adversarial samples, and then store the adversarial samples that meet the conditions in the adversarial sample library 22.

[0040] The raw sample data in the seed sample knowledge base 21 may include collected existing content security adversarial samples of large language models, known attack payloads, and data such as policy documents, regulations, and standards related to the normative use of large language models. For example, with user authorization, relevant statements or content generated by users interacting with the server 15 through the first terminal device 11, the second terminal device 12, and the third terminal device 13 can be collected as part of the raw sample data. Of course, the seed sample knowledge base 21 can also collect raw sample data from other sources. The raw sample data can be stored in the seed sample knowledge base 21 in the form of documents, with each document containing one or more sentences. A document may, for example, originate from a question or input during an interaction with any large language model, or it may be a collected independent file.

[0041] In some embodiments, the raw sample data collected in the seed sample knowledge base 21 can be preprocessed, and the preprocessed information can be stored in the seed sample knowledge base 21. This preprocessing includes, but is not limited to, vectorization, topic word extraction, keyword extraction, sentence extraction, and sentence dependency analysis. This preprocessed information can be selectively used as needed during the adversarial example generation process.

[0042] The action space 23 of the reinforcement learning algorithm can pre-define one or more candidate actions (also called "mutation operators") for generating adversarial examples. These candidate actions may include, but are not limited to, the character-level substitution, word-level substitution, and synonym generation methods mentioned above. In one embodiment, the pre-definement of one or more candidate actions in the action space 23 may include at least one of the following: synonym substitution, word order adjustment, candidate word insertion, candidate word deletion, and synonym generation. In the reinforcement learning algorithm, each mutation can select one or more actions from the action space 23 and combine them to obtain the target action. Then, the target sentence is mutated according to the target action to generate adversarial examples. The action space 23 pre-defines various mutation operators, which can generate adversarial examples from multiple dimensions such as vocabulary and sentences, increasing the richness and variability of adversarial examples.

[0043] The reward mechanism of reinforcement learning algorithms can be set to evaluate the quality of generated adversarial examples from at least one dimension, such as semantic similarity and fluency, to obtain a reward value. This allows for the continuous optimization of generated adversarial examples using the reward mechanism in reinforcement learning, improving semantic similarity and fluency, thereby enhancing the practicality and effectiveness of the adversarial examples.

[0044] It should be noted that the reinforcement learning-based adversarial example generation method and apparatus of the present invention can be used in the field of fintech, or in any field other than fintech. The present invention does not limit the application field.

[0045] The following will be based on Figure 1 Application scenarios and Figure 2 The system architecture, through Figures 3-8 The generation of adversarial examples based on reinforcement learning according to embodiments of the present invention will be described in detail.

[0046] Figure 3 A flowchart illustrating a reinforcement learning-based adversarial example generation method according to an embodiment of the present invention is shown.

[0047] like Figure 3 As shown, the reinforcement learning-based adversarial example generation method of this embodiment may include operations S310 to S350.

[0048] First, in operation S310, the target statement is obtained. For example, the target statement to be used to generate adversarial examples is extracted from the seed sample knowledge base 21.

[0049] Next, by operating S320~S340, the target statement is subjected to multiple rounds of iterative mutation based on the reinforcement learning algorithm to generate adversarial examples until the iteration termination condition is met.

[0050] Specifically, in operation S320, a first adversarial example is generated in each iteration based on the target action selected from the action space 23 of the reinforcement learning algorithm.

[0051] Then, in operation S330, the first adversarial example is evaluated from at least one first dimension to obtain a reward value, wherein the at least one first dimension includes: at least one of the semantic similarity between the first adversarial example and the target statement and the fluency of the first adversarial example; the reward value is positively correlated with both the semantic similarity and the fluency. In this way, reinforcement learning can proceed in the direction of continuously improving the semantic similarity and / or fluency of adversarial examples, thereby improving the quality of the adversarial examples.

[0052] In one embodiment, at least one first dimension further includes a diversity penalty factor, which is positively correlated with the repetition of the first adversarial sample with adversarial samples generated in historical iterations. That is, if a newly generated adversarial sample repeats a historical adversarial sample, the diversity penalty factor takes a larger value; otherwise, it takes a smaller value (e.g., 0). The reward value is negatively correlated with the diversity penalty factor. In this way, a diversity penalty factor can be introduced into the reward mechanism of reinforcement learning to deduct the reward value, thereby reducing the generation of duplicate adversarial samples during reinforcement learning and improving the efficiency and directionality of reinforcement learning.

[0053] Next, in operation S340, determine whether the iteration termination condition is met. If not, return to operation S320 for the next iteration. If the condition is met, terminate the iteration and execute operation S350.

[0054] The iteration terminates when at least one of semantic similarity or fluency stops improving. In this way, reinforcement learning continuously improves the fluency or semantic similarity of adversarial examples until it reaches its maximum, at which point the iteration terminates.

[0055] In operation S350, preferred adversarial samples that meet the first condition are selected from the generated adversarial samples. The first condition can be a condition determined based on at least one of the aforementioned first dimensions. For example, a threshold condition that must be met for semantic similarity or fluency. Or a condition that must be met for a diversity penalty factor, such as selecting all non-repeating adversarial samples based on the diversity penalty factor. Alternatively, the first condition can be to obtain a quality assessment value by weighting each adversarial sample based on the value of at least one first dimension, and then sorting them according to this quality assessment value, selecting the top k.

[0056] As can be seen, the embodiments of the present invention can utilize the reward mechanism in reinforcement learning to continuously improve the semantic similarity or fluency of adversarial examples, thereby improving the quality and attack effect of the final adversarial examples.

[0057] In one embodiment, obtaining the target statement in operation S310 may involve obtaining a first vectorized representation of the target statement, so that the reinforcement learning algorithm in operations S320 to S340 processes the target statement based on the first vectorized representation.

[0058] Figure 4 The flowchart illustrating the process of obtaining the first vectorized representation of a target statement in operation S310 of an adversarial example generation method according to an embodiment of the present invention is shown. Figure 4 As shown, according to this embodiment, operation S310 may include operations S401 to S403.

[0059] In operation S401, the first word embedding vector of each word in the target sentence is obtained. The conversion from words to word embedding vectors can be done using any conversion method, such as using a natural language processing tool that converts words into numerical vectors, or generating them through a large model with fine-tuning. For sentences in the seed sample knowledge base 21, the method for obtaining the first word embedding vector of each word can be determined by the preprocessing method used in the construction of the seed sample knowledge base 21.

[0060] In operation S402, the weight of each word is determined based on the importance category of each word in the target sentence. The importance category of each word can be obtained through the preprocessing process of the seed sample knowledge base 21.

[0061] In operation S403, the first word embedding vectors of the words in the target sentence are weighted and averaged according to their corresponding weights to obtain the first vectorized representation of the target sentence.

[0062] The higher the importance of a word, the higher its corresponding weight can be. This allows the word embedding vectors of highly important words in a sentence to be highlighted through weighting, making the first vectorized representation of the target sentence as close as possible to simulating the human understanding bias of different words when reading the target sentence. In this way, when the reinforcement learning algorithm mutates based on the first vectorized representation of the target sentence, it can, to some extent, make the adversarial examples also closer to the situation of human reading sentences.

[0063] In one embodiment, the importance category of each word in operation S402 can be determined based on its importance to the semantics of the sentence, such as whether it is an important part of the semantics (e.g., subject, predicate, and object). In other embodiments, a large language model can be selected to analyze whether the absence or replacement of a word in the sentence would cause the large language model to misunderstand the sentence, thereby determining the importance of the word. In still other embodiments, the importance category can be set to at least two categories: topic words and other words, where topic words are more important than other words. Furthermore, other words can be further divided into categories such as keywords and general words based on characteristics such as word frequency, where keywords are more important than general words.

[0064] When the importance category is set to at least two classes, namely subject terms and other words, it is necessary to distinguish whether each word is a subject term. In one embodiment, it can be determined whether each word in the target sentence belongs to the corresponding subject term set; if so, the word is a subject term; otherwise, the word is another word. In this embodiment, the subject term set needs to be extracted in advance. According to an embodiment of the present invention, the subject term set can be obtained by preprocessing the original sample data in the seed sample knowledge base 21. There are various ways to extract subject terms; one embodiment can be referred to. Figure 5 The illustration.

[0065] Specifically, refer to Figure 5 The original sample data in the seed sample knowledge base 21 can be clustered to obtain clusters. Each cluster is assumed to share a common topic, and the topic words for each cluster are extracted to obtain the corresponding topic word set, such as... Figure 5 In this study, keyword sets Z1, Z2, ..., Zc are extracted for each cluster L1, L2, ..., Lc. Extracting keywords from a cluster can improve the applicability and objectivity of the keywords.

[0066] Thus, the above operation S402 may include: when the target statement is extracted from the first document in the seed sample knowledge base 21, obtaining the set of subject words of the first cluster to which the first document belongs; for any first word in the target statement, when the first word belongs to the set of subject words of the first cluster, determining the importance category of the first word as a subject word, and when the first word does not belong to the set of subject words of the first cluster, determining the importance category of the first word as other words, wherein the weight corresponding to the subject word is greater than the weight corresponding to other words.

[0067] Figure 5 There are various ways to extract the keyword set for each cluster by operating S512. One implementation is as follows: Figure 6 As shown.

[0068] Specifically, in combination Figure 5 and Figure 6 Operation S512 may include operations S601 to S604.

[0069] First, in operation S601, the document vector of each document in the first cluster is obtained, wherein the document vector of each document is obtained by aggregating the second word embedding vectors of the words in the document.

[0070] The first cluster is the cluster containing the first document to which the target statement belongs. The method for extracting the keyword set is similar for other clusters.

[0071] The second word embedding vector of the words in the document can be the vector obtained when the words are transformed during the construction of the seed sample knowledge base 21.

[0072] The method for obtaining the second word embedding vector can be the same as or different from the method for obtaining the first word embedding vector used by the reinforcement learning algorithm in operation S401 when processing the target sentence. The construction and preprocessing of the seed sample knowledge base 21 are two different stages from the reinforcement learning algorithm's processing of the target sentence. When processing the target sentence, the reinforcement learning algorithm can reuse the word embedding vectors used in the preprocessing of the seed sample knowledge base 21 (if the seed sample knowledge base 21 stores the word embedding vector for each word), or it can convert and obtain the corresponding word embedding vectors in real time during the execution of the reinforcement learning algorithm.

[0073] The document vector for each document is obtained by aggregating the second word embedding vectors of the words in that document. In one embodiment, the document vector can have approximately the same length as the second word embedding vector. This reduces the dimensionality of the document vector, facilitating subsequent processing.

[0074] Next, in operation S602, the average of the document vectors of all documents in the first cluster is calculated to obtain the topic vector of the first cluster.

[0075] Then, in operation S603, from all the words in the first category, the second word embedding vectors whose distance from the topic vector satisfies the second condition are selected.

[0076] Finally, in operation S604, the words corresponding to the selected second word embedding vectors are used as topic words to obtain the topic word set.

[0077] As can be seen from the definition of topic vector, a topic vector can represent the center of a cluster (referred to as the cluster center). Therefore, the embodiments of this disclosure can determine whether a word is a topic word based on its distance from the cluster center, which can improve the accuracy and objectivity of topic word extraction.

[0078] The second condition could be that the distance to the topic vector does not exceed a preset threshold distance, thus selecting words whose distance to the topic vector is less than the preset threshold distance as topic words. In this case, the number of topic words in the topic word set is uncertain, and could even be zero. Alternatively, the second condition could be that words are sorted from closest to furthest from the topic vector, and the top-ranked words are selected as topic words. In this case, the number of topic words can be determined beforehand.

[0079] When there are multiple keywords in the keyword set, these keywords can be differentiated in importance based on their distance from the topic vector, with closer keywords having higher importance. Based on the importance distinction of keywords, different weights can be assigned to words identified as keywords in operation S402. For example, keywords in the keyword set can be sorted according to their importance. Then, when the first keyword belongs to the keyword set, it is assigned a corresponding weight based on its order within the keyword set. Thus, two keywords in the target statement can be assigned different weights based on their order within the keyword set. That is, keywords closer to the topic vector are more important in the vectorization of the target statement and have a higher weight in the vectorized representation of the target statement.

[0080] In the above operation S601, the document vector is obtained by aggregating the second word embedding vectors of words in the document. There are various aggregation methods. For example, the second word embedding vectors of words in the document can be summed and averaged, so that the length of each document vector is the same as the length of the second word embedding vector of a single word, reducing dimensionality. Another example is to avoid information loss during the aggregation of second word embedding vectors of words in the document into document vectors. A large language model can be used to first interpret the document's topic from multiple perspectives, and then the interpreted topic information can be combined with the document itself to transform and extract the document vector. One embodiment can be referred to below. Figure 7 As shown.

[0081] Figure 7 The flowchart illustrating operation S601 for obtaining document vectors of documents in a seed sample knowledge base is shown in one embodiment of the present invention.

[0082] like Figure 7 As shown, according to this embodiment, operation S601 may include operations S701 to S704.

[0083] In operation S701, for any second document, different first prompt words are used to guide the first large language model to interpret the second document, so as to obtain the first summary sentence output by the first large language model corresponding to each first prompt word. The first large language model can be any large language model.

[0084] Different first prompt words are used to handle different tasks. These different tasks include, but are not limited to, text classification tasks, sentiment analysis tasks, paraphrasing tasks, or information extraction tasks.

[0085] In operation S702, the second document is copied and a first summary sentence corresponding to each first prompt word is inserted into each second document to form a second document copy corresponding to each first prompt word. Thus, a second document copy is formed for each different first prompt word.

[0086] In operation S703, the second embedding vector of each word in the second document copy is obtained;

[0087] In operation S704, the second embedding vectors of all words in the second document copy are aggregated to obtain the document vector of the second document.

[0088] This approach first interprets the document from multiple perspectives, then merges the summarized statements obtained from the interpretation into the document before extracting the document vector. This reduces the loss of document topic information during the document vector conversion process. Moreover, interpreting from multiple perspectives and merging the interpretation information before extracting the document vector helps to retain more comprehensive topic information, thereby improving the comprehensiveness and objectivity of subsequent processing based on document vectors (clustering, keyword extraction, etc.).

[0089] The above has been approved. Figures 4-7 This paper introduces various implementations related to obtaining the first vectorized representation of a target sentence in reinforcement learning algorithms. In the process of obtaining the first vectorized representation of the target sentence, various information preprocessed during the construction of the seed sample knowledge base 21 (such as a topic word set and keyword set) can be utilized to improve the quality of the first vectorized representation of the target sentence. The more accurate and objective the first vectorized representation of the target sentence, and the closer it is to human reading habits, the more favorable the prerequisites for the reinforcement learning algorithm to process the target sentence and generate higher-quality adversarial examples.

[0090] Furthermore, in the reinforcement learning-based adversarial example generation method of this invention, the diversity, complexity, and variability of the generated adversarial examples can be improved by setting the diversity of candidate actions in the action space 23. The candidate actions in the action space 23 include, but are not limited to: synonym replacement, word order adjustment, candidate word insertion, candidate word deletion, and synonym sentence generation. Synonym replacement, word order adjustment, candidate word insertion, and candidate word deletion generate adversarial examples from a lexical perspective, while synonym sentence generation generates adversarial examples from a sentence perspective. For candidate actions such as synonym replacement, word order adjustment, candidate word insertion, and candidate word deletion, implementation can be aided by the dependency syntax tree of the target sentence and the importance categories of words. For example, priority can be given to mutating words that are not topic words or keywords, or to words with weak dependencies on other words in the sentence. This improves the targeting and directionality of lexical-based mutation, reduces semantic jumps in adversarial examples, reduces trial-and-error costs in reinforcement learning, and improves training efficiency. A specific embodiment can be referred to. Figure 8 As shown.

[0091] Figure 8 The diagram illustrates the operation flowchart of generating adversarial examples based on target actions in one iteration of the adversarial example generation method of another embodiment of the present invention.

[0092] like Figure 8 As shown, according to this embodiment, when the action selected from the action space includes at least one of word order adjustment or candidate word deletion, operation S320 may include operations S801 to S803.

[0093] In operation S801, the dependency syntax tree of the target sentence is obtained. In the dependency syntax tree, each word has a parent node, indicating that the word depends on its parent node. The entire tree ultimately forms a hierarchical structure with the "root node" at the top. The root node is the word in the entire dependency syntax tree that has no parent node and is the core of the entire sentence. Based on the dependency syntax tree, the type of dependency relationship between each word can be determined, including subject-verb, verb-object, attributive-head, and adverbial-head relationships. Finally, the strength of the dependency relationship can be quantified by the type of dependency relationship and the distance between the child node and the parent node.

[0094] The dependency syntax tree of the target statement can be obtained in the following way: based on the part-of-speech tagging of each word in the target statement, obtain the part-of-speech (e.g., noun, verb, adjective) of each word in the target statement; based on the part-of-speech of the words in the target statement, construct the dependency syntax tree, for example, by training a graph neural network model to construct the dependency syntax tree.

[0095] The dependency syntax tree of the target statement can be constructed in real time during the mutation process, or it can be preprocessed and stored during the construction of the seed sample knowledge base 21.

[0096] In operation S802, based on the strength of dependency relations in the dependency syntax tree of the target statement, words that satisfy the third condition are selected as the operation objects of the target action.

[0097] In operation S803, the target action is performed on the operation object to generate the first adversarial sample.

[0098] In the dependency syntax tree, the strength of the dependency relationship between two nodes is quantified by the distance between them and the type of dependency relationship traversed between them. For example, different weights can be assigned to different dependency relationship types. Subject-verb and verb-object relationships can be assigned higher weights, indicating a stronger dependency relationship, while attributive-head and adverbial-head relationships can be assigned lower weights. For instance, if two child nodes share a common parent node, the distance between them can be two edges (e.g., if quantized as 1, the distance is 2). These edges are then weighted according to the type of dependency relationship represented by each edge to obtain a specific value that characterizes the strength of the dependency relationship between the two nodes (this can be called the dependency relationship strength).

[0099] When the action selected from the action space 23 includes a word order adjustment action, the third condition includes that the strength of the dependency relationship between the two words whose order is being adjusted does not exceed a first preset threshold strength.

[0100] When the action selected from the action space 23 includes candidate word deletion, the third condition includes that the strength of the dependency relationship between the deleted word and the root node of the dependency syntax tree does not exceed a second preset threshold strength.

[0101] In some embodiments, the third condition can also be set based on the importance category of words in the sentence. For example, words undergoing mutation processing may not belong to topic words and keywords, thereby avoiding excessive semantic changes in the generated adversarial examples.

[0102] In this way, incorporating the dependency tree of the target statement and / or information on the importance categories of words can improve the targeting and directionality of word order adjustment and candidate word deletion actions in reinforcement learning, reduce the trial-and-error cost in reinforcement learning, and improve the efficiency and effectiveness of mutation.

[0103] As can be seen, the adversarial example generation method based on reinforcement learning provided in this embodiment of the invention can generate high-quality and stable adversarial examples through multiple rounds of iterative mutation. Furthermore, it can generate a massive number of adversarial examples while ensuring strong contextual relevance, grammatical fluency, and high semantic similarity of the adversarial example sentences. According to embodiments of the invention, based on the collected existing adversarial example datasets and related documents, preprocessing operations such as topic words, keywords, and dependency analysis can be performed on the dataset to construct a seed sample knowledge base 21. Then, a candidate sample set and synonym sentences are generated using mutation operators and a trained deep learning network. Based on this, the reward mechanism in the reinforcement learning algorithm is used to continuously improve semantic similarity and fluency. Ultimately, a massive number of adversarial examples are generated while ensuring strong contextual relevance, grammatical fluency, and high semantic similarity of the sentences.

[0104] The above describes various embodiments of the adversarial example generation method based on reinforcement learning according to the present invention. The following will illustrate... Figures 9-13 This paper introduces embodiments of an adversarial example generation device that can implement the above-described method. It is understood that... Figures 9-13 The apparatus shown is merely an example and does not constitute any limitation on the present invention.

[0105] Figure 9 The diagram schematically illustrates the structure of a reinforcement learning-based adversarial example generation apparatus 90 according to an embodiment of the present invention. This apparatus 90 integrates a reinforcement learning algorithm, a mutation operator, and a deep learning neural network, and can be used to generate adversarial examples.

[0106] Specifically, the device 90 may include an original sample preprocessing module 1, a candidate sample data generation module 2, and a candidate sample data evaluation module 3.

[0107] The original sample preprocessing module 1 can be as follows: Figure 10As shown, this is used to preprocess the raw sample data and build a seed sample knowledge base after collecting adversarial examples and related documents.

[0108] Candidate sample data generation module 2, as shown Figure 11 As shown, it is used to generate candidate adversarial samples based on the original sample preprocessing module 1, using mutation operators and deep learning neural networks.

[0109] Candidate Sample Data Evaluation Module 3, such as Figure 12 As shown, this method is used to improve the semantic similarity and fluency of candidate adversarial samples by leveraging the reward mechanism in reinforcement learning algorithms, and then the preferred adversarial samples that meet the conditions are stored.

[0110] Figure 10 The diagram illustrates the structure of the raw sample data preprocessing module 1 in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0111] like Figure 10 As shown, the original sample preprocessing module 1 may include a seed sample knowledge base construction unit 101 and a seed sample preprocessing unit 102. The main function of the seed sample knowledge base construction unit 101 is to collect adversarial samples and related policy documents related to the content security of the large language model, and then use this result as input to the seed sample preprocessing unit 102. After data preprocessing, keyword extraction and dependency parsing by the seed sample preprocessing unit 102, the data is stored in the seed sample knowledge base 21.

[0112] The seed sample knowledge base construction unit 101 is used to: collect existing adversarial samples and known attack payloads that attack large language models to ensure the dataset has broad representativeness; based on this, relevant policy documents, laws and regulations, and other standard documents can also be collected to form raw sample data, which is then stored as a document set. Let the document set be... The number of documents in the collection is Store the results in the seed sample knowledge base 21 and mark the source of each document.

[0113] The seed sample preprocessing unit 102 is used to extract subject terms, keywords, and sentence dependency relationships based on the document set and sources collected by the seed sample knowledge base construction unit 101.

[0114] In one embodiment, the seed sample preprocessing unit 102 may perform a reference... Figures 5-7 The described method is used for keyword extraction.

[0115] Specifically, a pre-trained large model can be used to embed and encode the documents. One embodiment involves the following steps: First, first prompt words are written for four tasks: text classification, sentiment analysis, paraphrasing, and information extraction. By covering four application scenarios of sentences, the first large language model is guided to understand sentences from specific perspectives. The first prompt word for each task guides the first large language model to generate an independent sentence; that is, each document will generate four first summary sentences from different perspectives.

[0116] After obtaining four distinct first summary sentences, a pre-trained large model can be used to embed the document. The specific steps are as follows:

[0117] Let any document in the document collection be... First, copy the same document four times. Then, insert a summary sentence for each of the four task types at the beginning of the document. The processed document is denoted as [document name missing]. Suppose that the content of any document can be represented by a vocabulary set. It means that among them This represents the number of words in the document. First, a pre-trained large model is used to convert each word into a second-word embedding vector. Then, document vectors are generated by aggregating the second embedding vectors of all words. The specific formula is as follows:

[0118]

[0119] After converting each document into a document vector, the Uniform Manifold Approximation and Projection (UMAP) algorithm can be used to perform nonlinear dimensionality reduction on the document vectors, effectively mapping high-dimensional data to a low-dimensional space. Let the dimensionality-reduced document vector be... Building upon this, a density-based hierarchical clustering algorithm (Hierarchical Density-Based Spatial Clustering of Applications with Noise, HDBSCAN) can be used to perform density clustering on document vectors, forming... The document vector is divided into several clusters. Each cluster is considered a topic, and the average of all document vectors within the same cluster is calculated to obtain the topic vector for that cluster. Then, the document vector closest to this topic vector is selected. The words corresponding to the second word embedding vectors are used as the topic words of this cluster, thus obtaining the topic word set of this cluster (e.g., ...). Figure 5(As shown). Based on the keyword set of each cluster, the keywords and number of each document are finally obtained. The keywords of each document are marked and stored in the seed sample knowledge base 21 along with the document content.

[0120] Next, the seed sample preprocessing unit 102 can extract keywords for each document. Based on each document and documents in its cluster that share the same topic words, the keywords for each document can be determined using a term frequency-inverse term frequency (TIF) algorithm, thus obtaining a keyword set for each document. Determining the keywords for a document based on the first document and documents in the first cluster that share the same topic words avoids the keyword dilution problem caused by analyzing keywords in an unrelated set of documents (those without the same topic words).

[0121] For example, we can define the document collection under each keyword as follows: For document collections Documents in Use the following formula to calculate each word Importance in the document:

[0122]

[0123] in It is a word In the document The number of times it appears in;

[0124] It is the total number of times all words appear in document d.

[0125] Then, the inverse document frequency is calculated:

[0126]

[0127] in It is a collection of documents The total number of files in the file;

[0128] It is a collection of documents It contains words Documents The quantity. The specific calculation formula is as follows:

[0129]

[0130] Then, the word t in the document is calculated. The importance of The specific calculation formula is as follows:

[0131]

[0132] Each document mark The top n keywords.

[0133] Based on the above steps, the topic words and the top n most important keywords for each document are obtained and stored in the seed sample knowledge base 21 along with the document content.

[0134] Furthermore, the seed sample preprocessing unit 102 can also be used to perform dependency parsing on each sentence in the seed sample knowledge base 21, obtaining the dependency relationships between words and storing them in the seed sample knowledge base 21. The specific steps are as follows: First, each word in the sentence is tagged with its part of speech to determine its part of speech, such as noun, verb, adjective, etc. Then, a dependency parsing tree is constructed based on the part of speech of the words. In the dependency parsing tree, each word has a parent node, indicating that the word depends on its parent node. Ultimately, the entire tree will form a hierarchical structure with the "root node" at the top. The root node is the word in the entire dependency parsing tree that has no parent node and is the core of the entire sentence. Based on the dependency parsing tree, the type of dependency relationship between each word can be determined, including subject-verb, verb-object, attributive-head, and adverbial-head relationships. Finally, the strength of the dependency relationship can be quantified by the type of dependency relationship and the distance between the child node and the parent node, and then stored in the seed question knowledge base.

[0135] Figure 11 The diagram illustrates the structure of the candidate sample data generation module 2 in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0136] like Figure 11 As shown, the candidate sample data generation module 2 may include a mutation operator candidate sample generation unit 201 and a synonym sample generation unit 202. The main function of the mutation operator candidate sample generation unit 201 is to generate new adversarial samples from a lexical perspective by combining contextual relationships with mutation operators. The synonym sample generation unit 202 mainly uses a fine-tuned deep learning network model to generate new adversarial samples from a sentence perspective.

[0137] The candidate sample data generation module 2 can be integrated into the action space 23 of the reinforcement learning algorithm to provide one or more candidate actions for generating adversarial examples.

[0138] The mutation operator candidate sample generation unit 201 can be used to implement sentence vectorization.

[0139] For example, firstly, a natural language processing tool that converts words into numerical vectors is used to vectorize the target sentence extracted from the seed sample knowledge base, obtaining the first vectorized representation of the target sentence. Specifically, each word can be mapped to a high-dimensional vector space to obtain the first word embedding vector. Let the word in a sentence be... Word The first word embedding vector is , where u is the embedding dimension.

[0140] For sentences Sentence vectorization can be achieved using a weighted average method. The specific formula is as follows:

[0141]

[0142] in As for weights, the weights of word vectors other than keywords and topic words can be set to 1. Let the initial weight of topic words be set to... The initial weight of the keywords is set to ,in, and All are greater than 1, and Greater than Scores can be assigned to keywords based on the distance between them and the topic vectors during preprocessing, and scores can also be assigned based on the importance of the keywords. A score is assigned to each keyword to differentiate the importance of different topic terms and keywords. For topic terms and keywords with different scores, the actual weights can be adjusted based on the scores assigned in the seed sample preprocessing unit 102, in addition to the initial weights, to further distinguish importance.

[0143] Subsequently, the mutation operator candidate sample generation unit 201 can also construct four mutation operators: a synonym substitution operator, a word order adjustment operator, a candidate word insertion operator, and a candidate word deletion operator. These four mutation operators can be associated with or integrated into the action space 23 as candidate actions.

[0144] For the synonym substitution operator, for sentences Each word We can simultaneously search for its synonym set from both vector and lexical perspectives. Let the synonym set be... Generate new sentences The specific formula is as follows:

[0145]

[0146] The set of synonyms It can be dynamically generated by a large language model. The specific generation steps are as follows: Based on the topic words, keywords and dependency analysis results obtained in the seed sample preprocessing unit 102, prompt words are designed to guide the large language model to output a set of thematic and context-related synonyms and store them.

[0147] For the word order reordering operator, based on the dependency syntax tree obtained in the seed sample preprocessing unit 102, words with weaker relationships can be prioritized for word order swapping to ensure the integrity of the sentence structure. Let the two weakest words be denoted as... and After swapping the order, a new sentence is finally generated. .

[0148] For the candidate word insertion operator, predicates and noun phrases in a sentence can be identified based on the dependency syntax tree obtained in the seed sample preprocessing unit 102. Candidate words can then be inserted preferentially between noun phrases or after verbs to maintain the integrity of the grammatical structure. The method for generating candidate words is as follows: First, based on the embedding vectors of the first word of the preceding and following words, words whose cosine similarity satisfies certain conditions (e.g., selecting the highest cosine similarity) are found in the vector space as the set of insertion candidate words.

[0149] Let the inserted new vocabulary be... The insertion position is the first one. The word and the first Generate new sentences from words .

[0150] For the candidate word deletion operator, based on the dependency syntax tree, keywords, and subject terms obtained from the seed sample preprocessing unit 102, words with weak dependency relationships and not belonging to keywords and subject terms, such as adverbs or adjectives, can be prioritized for deletion to maintain the core meaning of the sentence. Let the selected deletion word be... Generate new sentences .

[0151] The synonym sample generation unit 202 uses the collected original sentence-similar sentence pairs as training samples and manually labels positive and negative sample pairs, where positive samples are similar sentences and negative samples are random samples or other negative examples. Based on the above samples, the second language model is fine-tuned and trained so that the second language model can generate synonyms from the sentence perspective.

[0152] Figure 12 The diagram illustrates the structure of the candidate sample data evaluation module 3 in an adversarial sample generation apparatus according to an embodiment of the present invention.

[0153] like Figure 12 As shown, the candidate sample data evaluation module 3 may include a sample quality enhancement unit 301 and a sample storage unit 302.

[0154] The sample quality enhancement unit 301 continuously enhances the quality of new sentences (i.e., adversarial examples) generated by the mutation operator candidate sample generation unit 201 and the synonym sample generation unit 202 through a reinforcement learning reward mechanism, thereby improving semantic similarity and fluency. The sample storage unit 302 stores adversarial examples that meet the requirements. In one embodiment, the candidate sample evaluation module 3 can perform the operations S310 to S350 described above.

[0155] Specifically, the sample quality enhancement unit 301 uses reinforcement learning combined with the mutation operator in the candidate sample data generation module 2 to improve the fluency and semantic similarity of the generated adversarial samples.

[0156] First, we can define the environment state, action space, reward mechanism, and discount factor in reinforcement learning. The environment state uses the target statement. The first vector representation of the action space (e.g., ...). Figure 2 The action space 23 can contain the four mutation operators in the mutation operator candidate sample generation unit 201 and the synonym generation operator in the synonym sample generation unit 202. That is, candidate actions can include five actions: synonym replacement, word order adjustment, candidate word insertion, candidate word deletion, and synonym generation. In each iteration of reinforcement learning, one or more actions can be selected from the action space 23 and combined to form the target action. In other words, when generating a new sentence, in addition to the case where the five mutation operators (synonym replacement, word order adjustment, candidate word insertion, candidate word deletion, and synonym generation) generate new sentences independently as shown in the previous example, any number of mutation operators can be combined to generate new sentences to obtain adversarial examples.

[0157] In reinforcement learning, a new sentence is generated in each iteration based on the selected target action. (i.e., adversarial examples), and the reward value can be calculated according to the following formula. :

[0158]

[0159] in To measure the semantic similarity between the generated sentence and the target sentence, To improve the fluency of the generated sentences, It serves as a diversity penalty factor.

[0160] In one embodiment, semantic similarity

[0161] The calculation formula is as follows:

[0162]

[0163] in, To generate sentences The vector, For target statement The vector is .

[0164] In one embodiment, smoothness The calculation formula is as follows:

[0165]

[0166] in The perplexity is calculated using a mature language model. The above formula is used to calculate fluency. (The sentence length is not specified.) At that time, the perplexity was normalized using sentence length.

[0167] Through diversity penalty factor If the generated sentence is identical to a previous sentence, the diversity penalty factor will be positive or increase, thus incurring a penalty. If the generated sentence is not identical to a previous sentence, the diversity penalty factor will decrease or even become zero.

[0168] In the process of reinforcement learning, a proximal policy optimization algorithm can be used to optimize the policy until the semantic similarity or fluency of the generated new sentences (adversarial examples) no longer improves.

[0169] The generated sample storage unit 302 is used to process the generated adversarial samples according to the reward value. The adversarial samples are sorted by size, and the Top-K samples are stored as preferred adversarial samples. The number of adversarial samples before and after generation can also be statistically analyzed.

[0170] Figure 13 The schematic diagram illustrates the workflow of a reinforcement learning-based adversarial sample generation device 90 according to an embodiment of the present invention.

[0171] like Figure 13 As shown, the workflow of device 90 includes steps S1 to S7.

[0172] Step S1: The seed sample knowledge base construction unit 101 in the raw data preprocessing module 1 collects existing raw sample data such as adversarial samples and initially constructs the seed sample knowledge base 21.

[0173] Step S2: The seed sample preprocessing unit 2 in the raw data preprocessing module 1 preprocesses the raw sample data in the sample knowledge base 11, including topic word extraction, keyword extraction, and dependency syntax tree construction.

[0174] Step S3: The mutation operator candidate sample generation unit 201 in the candidate sample data generation module 2 generates the target sentence from a lexical perspective based on the preprocessing results in step 2.

[0175] Step S4: The synonym sample generation unit 202 in the candidate sample data generation module 2 generates synonyms of the target sentence from the sentence perspective based on the preprocessing results in step 2.

[0176] Step S5: The sample quality enhancement unit 301 of the candidate sample data evaluation module 3 optimizes the actual execution of the mutation algorithm in steps S3 and S4 by configuring reinforcement learning algorithm, so as to improve the semantic similarity and fluency of the generated adversarial samples.

[0177] Step S6: Determine whether the semantic similarity and fluency of the generated new sentence meet the requirements through a reinforcement learning reward mechanism. If the semantic similarity and fluency meet the requirements, proceed to Step 7 to store the results. If the above conditions are not met, proceed to Step 2, reprocess the seed sample knowledge base, continue to learn and adjust the contextual relationships, topic words, keywords, and other features of the original sample data before generating samples again.

[0178] Step S7: The generated sample storage unit 302 of the candidate sample data evaluation module 3 is responsible for storing the preferred adversarial samples that meet the conditions after generation.

[0179] As can be seen, in the process of constructing the seed sample knowledge base 21, the embodiments of the present invention extract topic words, keywords and dependency syntax trees, so that the adversarial sample generation process can better capture the sample context relationship and improve the generation quality and diversity of adversarial samples.

[0180] In this embodiment of the invention, mutation operators such as synonym replacement, word order adjustment, candidate word insertion, candidate word deletion, and synonym sentence generation can generate adversarial samples from both lexical and sentence perspectives, increasing the richness and variability of adversarial samples.

[0181] In this embodiment of the invention, the reward mechanism in reinforcement learning is used to continuously optimize the generated adversarial examples, improve semantic similarity and fluency, thereby enhancing the practicality and effectiveness of the adversarial examples.

[0182] According to embodiments of the present invention, any plurality of modules in the original sample preprocessing module 1, the candidate sample data generation module 2, and the candidate sample data evaluation module 3 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the original sample preprocessing module 1, the candidate sample data generation module 2, and the candidate sample data evaluation module 3 can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuit, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the original sample preprocessing module 1, the candidate sample data generation module 2, and the candidate sample data evaluation module 3 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0183] Figure 14 A block diagram schematically illustrates an electronic device suitable for implementing a reinforcement learning-based adversarial example generation method according to an embodiment of the present invention.

[0184] like Figure 14 As shown, an electronic device 1400 according to an embodiment of the present invention includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage portion 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0185] RAM 1403 stores various programs and data required for the operation of electronic device 1400. Processor 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Processor 1401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1402 and / or RAM 1403. It should be noted that the programs may also be stored in one or more memories other than ROM 1402 and RAM 1403. Processor 1401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0186] According to an embodiment of the present invention, the electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to a bus 1404. The electronic device 1400 may also include one or more of the following components connected to the input / output (I / O) interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1410 as needed so that computer programs read from it can be installed into the storage section 1408 as needed.

[0187] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0188] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the adversarial example generation method provided in the embodiments of the present invention.

[0189] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A reinforcement learning-based adversarial example generation method, comprising: Obtain the target statement; The target statement is subjected to multiple rounds of iterative mutation based on a reinforcement learning algorithm to generate adversarial examples until the iteration termination condition is met. Select the preferred adversarial samples that meet the first condition from the generated adversarial samples; The step of generating adversarial examples by performing multiple rounds of iterative mutation on the target statement based on reinforcement learning algorithms includes: In each iteration, a first adversarial example is generated based on a target action selected from the action space of the reinforcement learning algorithm; and The first adversarial sample is evaluated from at least one first dimension to obtain a reward value, wherein the at least one first dimension includes at least one of the semantic similarity between the first adversarial sample and the target statement and the fluency of the first adversarial sample; the reward value is positively correlated with both the semantic similarity and the fluency. The iteration termination condition includes at least one of the semantic similarity or the fluency no longer improving.

2. The method according to claim 1, wherein, The action space contains one or more preset candidate actions; Wherein, when only one action is selected from the action space, the target action is the selected action; and when multiple actions are selected from the action space, the target action is a combination of the selected actions. The one or more candidate actions include at least one of the following: synonym replacement, word order adjustment, candidate word insertion, candidate word deletion, and synonym sentence generation.

3. The method according to claim 1, wherein, At least one first dimension also includes a diversity penalty factor, which is positively correlated with the repetition of the first adversarial sample and the adversarial samples generated in historical iterations, wherein the reward value is negatively correlated with the diversity penalty factor.

4. The method according to claim 2, wherein, The step of generating the first adversarial sample in each iteration based on the target action selected from the action space of the reinforcement learning algorithm includes: When the action selected from the action space includes at least one of word order adjustment or candidate word deletion, generating a first adversarial example based on the target action selected from the action space of the reinforcement learning algorithm includes: Obtain the dependency syntax tree of the target statement; Based on the strength of dependency relations in the dependency syntax tree of the target statement, words that satisfy the third condition are selected as the operation objects of the target action. In the dependency syntax tree, the strength of the dependency relation between the words corresponding to the two nodes is quantified by the distance between the two nodes and the type of dependency relation between the two nodes.

5. The method according to claim 4, wherein, When the action selected from the action space includes word order adjustment, the third condition includes that the strength of the dependency relationship between the two words whose order is being adjusted does not exceed a first preset threshold strength. When the action selected from the action space includes candidate word deletion, the third condition includes that the strength of the dependency relationship between the deleted word and the root node of the dependency syntax tree does not exceed a second preset threshold strength.

6. The method according to claim 1, wherein, The acquisition of the target statement includes acquiring a first vectorized representation of the target statement, wherein the reinforcement learning algorithm processes the target statement based on the first vectorized representation; wherein the first vectorized representation of the target statement includes: Obtain the first word embedding vector for each word in the target statement; The weight of each word is determined based on the importance category of each word in the target sentence; The first word embedding vectors of the words in the target sentence are weighted and averaged according to their corresponding weights to obtain the first vectorized representation of the target sentence.

7. The method according to claim 6, wherein, The importance categories are divided into at least two categories: topic words and other words. The determination of the weight of each word based on its importance category in the target sentence includes: Determine the first document to which the target statement belongs in the seed sample knowledge base, wherein the target statement is extracted from the first document; Obtain the keyword set of the first cluster to which the first document belongs, wherein the first cluster is one of multiple clusters obtained by clustering the documents in the seed sample knowledge base, and the keyword set of the first cluster is obtained based on the words in the documents in the first cluster; For any first word in the target statement, when the first word belongs to the topic word set, the importance category of the first word is determined to be a topic word, and when the first word does not belong to the topic word set, the importance category of the first word is other words; Among them, the weight of the keyword is greater than the weight of other words.

8. The method according to claim 7, wherein, The set of keywords for the first cluster to which the first document belongs was determined in the following way: Obtain the document vector of each document in the first cluster, wherein the document vector of each document is obtained by aggregating the second word embedding vectors of the words in the document; The topic vector of the first cluster is obtained by averaging the document vectors of all documents in the first cluster. From all documents in the first cluster, select second word embedding vectors whose distance to the topic vector satisfies the second condition; and The words corresponding to the selected second word embedding vectors are used as topic words to obtain the topic word set.

9. The method according to claim 8, wherein, The step of obtaining the document vector for each document in the first cluster includes: For any second document, different first prompt words are used to guide the first language model to interpret the second document, so as to obtain the first summary sentence output by the first language model corresponding to each first prompt word. Different first prompt words are used to process different tasks, including at least one of the following: text classification task, sentiment analysis task, paraphrase recognition task, or information extraction task. Copy the second document and insert a first summary sentence corresponding to each first prompt word into each second document to form a copy of the second document corresponding to each first prompt word; Obtain the second embedding vector for each word in the second document copy; The second embedding vectors of all words in the second document copy are aggregated to obtain the document vector of the second document.

10. The method according to claim 7, wherein, The other words are divided into at least two categories: keywords and general words; wherein, determining the weight of each word based on its importance category in the target sentence further includes: Obtain the keyword set of the first document, wherein the keyword set of the first document is determined based on the first document and documents in the first cluster that have the same topic words as the first document, using a term frequency-inverse term frequency algorithm; and When the first word does not belong to the topic term set but belongs to the keyword set, the importance category of the first word is determined to be a keyword; otherwise, the importance category of the first word is determined to be a general word; wherein, the weight corresponding to the keyword is greater than the weight corresponding to the general word.

11. An adversarial example generation device based on reinforcement learning, wherein, The device includes: A sample quality enhancement unit is generated to acquire the target statement and perform multiple rounds of iterative mutation on the target statement based on a reinforcement learning algorithm to generate adversarial samples until the iteration termination condition is met. A sample storage unit is generated to select the preferred adversarial samples that meet the first condition from the generated adversarial samples; Specifically, the sample quality enhancement unit is used for: In each iteration, a first adversarial example is generated based on a target action selected from the action space of the reinforcement learning algorithm; and The first adversarial sample is evaluated from at least one first dimension to obtain a reward value, wherein the at least one first dimension includes at least one of the semantic similarity between the first adversarial sample and the target statement and the fluency of the first adversarial sample; the reward value is positively correlated with both the semantic similarity and the fluency. The iteration termination condition includes at least one of the semantic similarity or the fluency no longer improving.

12. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program or instructions stored thereon, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions, wherein, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

Images