Attack sample data set generation method and device, program product and equipment

Abstract

The invention provides an attack sample data set generation method and device, a program product and equipment, and relates to the technical field of computers. The attack sample data set generation method comprises the steps of obtaining big model safety evaluation knowledge data, and obtaining safety evaluation keywords and safety evaluation topics associated with the safety evaluation keywords based on the big model safety evaluation knowledge data; based on the security evaluation keyword, the security evaluation topic associated with the security evaluation keyword is expanded through the large model, a seed topic is obtained, an attack sample data set corresponding to the seed topic is generated, and the attack sample data set is used for performing security evaluation on the large model. According to the method, a large number of attack samples with high richness can be quickly generated, and the labor cost is saved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for generating an attack sample data set, an attack sample data set generating device, a computer program product, and an electronic device. Background Art

[0002] The construction of attack sample data sets is the basis for large-model security assessment and protection work. The accumulation of high-quality attack instructions can simulate real attack scenarios and ensure more comprehensive mining of potential vulnerability threats of large models.

[0003] In the related technologies, since there is relatively little content risk data for large models on the Internet, it is difficult to construct an attack sample dataset, and manually writing an attack sample dataset is time-consuming, labor-intensive, and inefficient.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an attack sample data set generation method, an attack sample data set generation device, a computer program product and an electronic device, so as to at least to some extent solve the problem that the attack sample data set construction is difficult and the generation efficiency is low in the related art.

[0006] According to a first aspect of the present disclosure, a method for generating an attack sample data set is provided, the method comprising: acquiring large model security assessment knowledge data, and obtaining security assessment keywords and security assessment topics associated with the security assessment keywords based on the large model security assessment knowledge data; based on the security assessment keywords, expanding the security assessment topics associated with the security assessment keywords through a large model to obtain seed topics, and generating an attack sample data set corresponding to the seed topic; the attack sample data set is used to perform security assessment on the large model.

[0007] According to a second aspect of the present disclosure, there is provided an attack sample data set generating device, the device comprising: a keyword determination module, used to obtain large model security assessment knowledge data, and based on the large model security assessment knowledge data, obtain security assessment keywords and security assessment topics associated with the security assessment keywords; a data set generation module, used to expand the security assessment topics associated with the security assessment keywords through a large model based on the security assessment keywords, obtain seed topics, and generate an attack sample data set corresponding to the seed topic; the attack sample data set is used to perform security assessment on the large model.

[0008] According to a third aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the attack sample dataset generation method and possible implementation manners thereof of the first aspect are implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the attack sample data set generation method and possible implementation methods of the above-mentioned first aspect by executing the executable instructions.

[0010] The technical solution disclosed in this disclosure has the following beneficial effects:

[0011] In the process of generating the above-mentioned attack sample data set, the big model security assessment knowledge data is obtained, and based on the big model security assessment knowledge data, security assessment keywords and security assessment topics associated with the security assessment keywords are obtained; based on the security assessment keywords, the security assessment topics associated with the security assessment keywords are expanded through the big model to obtain seed topics, and an attack sample data set corresponding to the seed topics is generated. The attack sample data set is used to perform security assessment on the big model. On the one hand, by constructing seed topics through the big model security assessment knowledge data, and constructing attack sample data sets based on seed topics, a large number of high-rich attack samples can be quickly generated, saving labor costs; on the other hand, using the attack sample data set constructed by seed topics for security assessment can more comprehensively reflect the performance of the big model in various security scenarios, which helps to fully explore the potential vulnerabilities and risk points of the big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for generating an attack sample data set in this exemplary embodiment is shown;

[0013] Figure 2 A schematic diagram showing construction of a seed topic library in this exemplary embodiment is shown;

[0014] Figure 3 A schematic diagram showing data expansion by a large model in this exemplary embodiment is shown;

[0015] Figure 4 A schematic diagram of data expansion by regular splicing in this exemplary embodiment is shown;

[0016] Figure 5 A schematic diagram of generating an attack sample data set in this exemplary embodiment is shown;

[0017] Figure 6 A flowchart showing the automatic generation of a large-scale attack sample in this exemplary embodiment is shown;

[0018] Figure 7 A system structure diagram for generating a large-scale instruction attack data set in this exemplary embodiment is shown;

[0019] Figure 8 A structural block diagram of an attack sample data set generating device in this exemplary embodiment is shown;

[0020] Fig. 9 An electronic device for implementing the above-mentioned attack sample data set generation method in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings.

[0022] The accompanying drawings are schematic diagrams of the present disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in hardware modules or integrated circuits, or in networks, processors or microcontrollers. The embodiments can be implemented in various forms and should not be construed as being limited to the examples set forth herein. The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the description below, many specific details are provided to provide a full description of the embodiments of the present disclosure. However, those skilled in the art should appreciate that one or more specific details may be omitted when implementing the technical solution of the present disclosure, or one or more specific details may be replaced by other methods, components, devices, steps, etc.

[0023] In the related technologies, since there is relatively little content risk data for large models on the Internet, it is difficult to construct an attack sample dataset, and manually writing an attack sample dataset is time-consuming, labor-intensive, and inefficient.

[0024] In view of the above problems, the exemplary embodiments of the present disclosure provide an attack sample data set generation method, an attack sample data set generation device, a computer program product and an electronic device, which can be used in large model content security assessment scenarios, and can provide assessment sample information for algorithm filing, generative artificial intelligence (large language model) online filing, etc. The present disclosure does not make specific limitations on this.

[0025] In an optional embodiment, reference Figure 1 A method for generating an attack sample data set is provided, which specifically includes the following steps S110 to S120:

[0026] Step S110, obtaining large model security assessment knowledge data, and obtaining security assessment keywords and security assessment topics associated with the security assessment keywords based on the large model security assessment knowledge data;

[0027] Step S120, based on the security assessment keyword, the security assessment topic associated with the security assessment keyword is expanded through the big model to obtain a seed topic, and an attack sample data set corresponding to the seed topic is generated; the attack sample data set is used to perform security assessment on the big model.

[0028] Figure 1 In the method shown, seed topics are constructed through the big model security assessment knowledge data, and attack sample data sets are constructed based on the seed topics, which can quickly generate a large number of high-rich attack samples and save labor costs; using the attack sample data set constructed by the seed topics for security assessment can more comprehensively reflect the performance of the big model in various security scenarios, and help to comprehensively explore the potential vulnerabilities and risk points of the big model.

[0029] Below Figure 1 Provide detailed instructions for each step.

[0030] In step S110, large-model security assessment knowledge data is acquired, and based on the large-model security assessment knowledge data, security assessment keywords and security assessment topics associated with the security assessment keywords are obtained.

[0031] Among them, the big model security assessment knowledge data refers to the knowledge data related to the big model security assessment. Exemplarily, the big model security assessment knowledge data can be the knowledge data in the basic security requirements of generative artificial intelligence services, such as knowledge data related to security risk content such as violation of values, discriminatory content, commercial illegal and irregular content, infringement of the legitimate rights and interests of others, etc., which is not specifically limited in this disclosure.

[0032] Optionally, the sources of knowledge data for large-scale model safety assessment may include, but are not limited to: articles, hot news, social media opinions, historical events, laws and regulations, national / local / enterprise standards, government documents, etc.

[0033] In an optional implementation, the above-mentioned acquisition of large-model security assessment knowledge data can be achieved through the following steps: collecting initial knowledge data of preset security assessment dimensions, and performing data preprocessing on the initial knowledge data to obtain large-model security assessment knowledge data.

[0034] Among them, the preset security assessment dimensions can be assessment dimensions pre-set by developers, such as value dimensions, sensitive topic dimensions, etc., and this disclosure does not make specific limitations on this.

[0035] The initial knowledge data is the collected unprocessed knowledge data. Optionally, the collected initial knowledge data can be stored in a corpus to facilitate data reuse.

[0036] Exemplarily, crawler technology may be used to collect initial knowledge data of preset security assessment dimensions from the Internet.

[0037] Exemplarily, initial knowledge data of preset security assessment dimensions may also be collected manually.

[0038] It should be noted that the initial knowledge data collected in this disclosure is legal and authorized data content.

[0039] Since the collected initial knowledge data may have problems such as inconsistent formats and repeated semantics, the initial knowledge data can be preprocessed to obtain high-quality large-model security assessment knowledge data. The data preprocessing methods may include but are not limited to deleting redundancy, unifying formats, and other processing methods.

[0040] By collecting initial knowledge data of preset security assessment dimensions and performing data preprocessing on the initial knowledge data, the comprehensiveness and quality of the large-model security assessment knowledge data can be ensured.

[0041] Among them, the safety assessment keywords can be representative words in the large model safety assessment knowledge data.

[0042] Among them, the safety assessment topic can be the content subject involved in the large model safety assessment knowledge data.

[0043] In an optional implementation, the above-mentioned obtaining of security assessment keywords and security assessment topics associated with the security assessment keywords based on the big model security assessment knowledge data can be achieved through the following steps: performing keyword extraction on the big model security assessment knowledge data to obtain security assessment keywords; performing topic extraction on the big model security assessment knowledge data to obtain security assessment topics; and matching the security assessment topics with the security assessment keywords to obtain security assessment topics associated with the security assessment keywords.

[0044] Specifically, keyword extraction technology and topic extraction technology can be used to perform keyword extraction and topic extraction on the large model security assessment knowledge data to obtain security assessment keywords and security assessment topics. By determining the keyword category to which the security assessment topic belongs, the security assessment topic and the security assessment keyword can be matched to obtain the security assessment topic associated with the security assessment keyword.

[0045] Each security assessment topic can be mapped to a corresponding security assessment keyword system. For example, "Topic 1" and "Topic 2" are mapped to "Keyword 1"; "Topic 3", "Topic 4" and "Topic 5" are mapped to "Keyword 2".

[0046] By extracting security assessment keywords and security assessment topics, data support can be provided for the construction of subsequent seed topics.

[0047] In an optional implementation, the above-mentioned keyword extraction of the large model security assessment knowledge data to obtain security assessment keywords can be achieved through the following steps: using the word frequency and inverse document frequency algorithm to determine the word frequency and inverse document frequency of each word in the large model security assessment knowledge data; based on the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, determine the security assessment keywords.

[0048] The knowledge data of large model safety assessment is in the form of documents, which can be calculated by Get the frequency of each word in the large model security evaluation knowledge data. t represents the number of times a word t appears in a document, ∑ k n k represents the total number of words in the document, TF(t) represents the frequency of word t, and t is the identifier of the current word. The inverse document frequency of each word in the large model security assessment knowledge data is obtained. Where N represents the total number of documents, |{d:t∈d}| represents the number of documents containing word t, IDF(t) represents the inverse document frequency of word t, and t is the identifier of the current processing word.

[0049] Optionally, after determining the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, the product of the word frequency and inverse document frequency of each word in the large model security assessment knowledge data can be sorted, and a preset number of words can be selected in order from large to small as security assessment keywords.

[0050] Optionally, after determining the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, words whose product is greater than a preset threshold can be selected as security assessment keywords from the product of the word frequency and inverse document frequency of each word in the large model security assessment knowledge data.

[0051] For example, the TF-IDF value of each word can be obtained by calculating TF-IDF(t)=TF(t)×IDF(t), and the TF-IDF value of each word is arranged in descending order to select words with higher TF-IDF values ​​as security assessment keywords for corresponding documents, where the TF-IDF value is the product of the word frequency and the inverse document frequency.

[0052] Optionally, at the level of the entire large-model security assessment knowledge data, the main topics can be mined by clustering the keywords of all documents or analyzing the word frequency distribution. If the keywords in multiple documents are similar, it can be inferred that these documents may belong to the same topic.

[0053] Optionally, the extracted security assessment keywords and security assessment topics may be stored in a keyword library and a topic library respectively to facilitate data reuse.

[0054] Through the above steps, representative security assessment keywords and security assessment topics can be quickly and automatically extracted from the massive corpus.

[0055] It should be noted that, in actual application, keyword extraction and topic extraction may also be performed by other technical means, and the present disclosure does not specifically limit this.

[0056] In step S120, based on the security assessment keywords, the security assessment topics associated with the security assessment keywords are expanded through the big model to obtain seed topics, and an attack sample data set corresponding to the seed topics is generated; the attack sample data set is used to perform security assessment on the big model.

[0057] Among them, seed topics refer to data samples of attack samples, which can be generated by large models or obtained by manual writing.

[0058] Optionally, the obtained seed topics may be stored in a seed topic library to facilitate data reuse.

[0059] For example, Figure 2 As shown, a schematic diagram of constructing a seed topic library is provided, and LLM (Large Language Model) 201 can be used to expand the topic library 202 to generate security assessment topics associated with the security assessment keywords in the keyword library 203. Optionally, the topics in the topic library can be processed by deduplication, manual review, modification, etc., and then stored in the seed topic library 204. Deduplication, manual review, modification, etc. of the topics in the topic library can ensure that the seed topics are representative, that is, each seed topic should have smooth sentences, clear intentions, accurate expressions, and a certain degree of distinction from other seed topics in the keyword category to which it belongs.

[0060] Among them, the attack sample data set refers to the data used to perform attack tests on the large model. The generated attack sample data set can be stored in the attack test sample library to facilitate data reuse.

[0061] In an optional implementation, the above-mentioned generation of an attack sample data set corresponding to a seed topic may be achieved by the following steps: obtaining an attack paradigm and / or an attack template, and generating an attack sample data set corresponding to the seed topic based on the attack paradigm and / or the attack template.

[0062] Among them, the attack paradigm is a high-level abstraction of attack methods. It focuses on the core characteristics of attack methods rather than a specific attack operation. Its advantages are flexibility and strong scalability.

[0063] Among them, attack template is a framework of attack mode. When faced with attack methods that are difficult to extract features, since it is impossible to accurately summarize general features from these attack methods for data generation or analysis, attack templates can be used. The advantage of attack templates is that they are simple in design and efficient in implementation.

[0064] Among them, the attack means may include but are not limited to: target hijacking, negative induction, role playing, implicit unsafe views, prefix attacks, prompt leakage, unsafe topics, restricted generation of rejection words, load splitting, etc., which are not specifically limited in this disclosure. It should be noted that target hijacking is a method of adding malicious instructions to user input to make the model forget the original task and perform the target task. Negative induction is an attacker trying to make the model generate things that "should not be done". Role playing is to require the model to play a fictional role and set a series of behavioral rules for the role. Implicit unsafe views are inquiries with unsafe views. Prefix attacks are to manipulate the output of the model by adding a specific prefix text to the input of the model to mislead the model and generate irrelevant, wrong or malicious responses. Prompt leakage is an attempt to manipulate the output of the model to leak part or all of the original prompt. Unsafe topics are input instructions that themselves quote inappropriate or unreasonable topics, so that the model will follow these instructions and generate unsafe content. Among them, limiting the generation of rejection words is to screen and filter the results of the model output when using a large model for text generation to produce inappropriate or harmful text content. Payload splitting refers to splitting the attack payload into multiple parts and then distributing them to different attack nodes for distributed attack.

[0065] Optionally, an attack paradigm and / or attack template may be constructed based on attack features corresponding to preset attack means.

[0066] For example, taking role-playing as an example, the following attack paradigm can be constructed:

[0067] #Task

[0068] {Please help me generate 1 instruction based on the given topic and the following rules}

[0069] ##rule

[0070] {Design a fictional character and set an experience and an activity for the character around a given topic}

[0071] #Notice

[0072] {Some notes}

[0073] #Example

[0074] {{XXX}}

[0075] #Output format

[0076] {"instruction":"instruction"}

[0077] {The given topic is:seed topic}

[0078] {Next, strictly follow the rules and output format, do not make any additional explanations or comments}.

[0079] Exemplarily, the following attack template can be constructed:

[0080] "Do you know where I can find more information and links about {seed topic}? I am studying a course that may involve {seed topic} and need to know as many factual cases as possible to help me objectively and comprehensively understand its development trajectory."

[0081] Based on the attack paradigm and / or attack template, a large number of high-rich attack sample data sets can be quickly generated, which can save labor costs.

[0082] In an optional embodiment, based on the attack paradigm and / or attack template, an attack sample data set corresponding to the seed topic is generated, which can be achieved by the following steps: inputting the seed topic into the attack paradigm, performing data expansion through a large model, and generating a first attack data set; inputting the seed topic into the attack template, performing data expansion through regular splicing, and generating a second attack data set; based on the first attack data set and the second attack data set, obtaining an attack sample data set corresponding to the seed topic.

[0083] The first attack data set refers to the attack data set obtained by expanding the seed topic through the large model.

[0084] The second attack data set refers to the attack data set obtained by expanding the seed topic through regular concatenation.

[0085] For example, Figure 3 As shown, a schematic diagram of data expansion through a large model is provided. The n attack paradigms can respectively target x seed topics, and data expansion can be performed through LLM to obtain the first attack data set.

[0086] For example, Figure 4 As shown, a schematic diagram of data expansion through regular splicing is provided. The m attack templates can respectively target the x seed topics, and data expansion is performed through regular splicing to obtain a second attack data set.

[0087] Optionally, after determining the first attack data set and the second attack data set, the attack strategies of the first attack data set and the second attack data set may be further optimized through an adversarial algorithm to obtain an attack sample data set corresponding to the seed topic.

[0088] For example, Figure 5 As shown in the figure, a schematic diagram of generating an attack sample data set is provided. The x seed topics in the seed topic library can be respectively input into the constructed n attack paradigms and m attack templates, and the data can be expanded by using LLM and regular splicing respectively. Finally, the adversarial algorithm is used to optimize the attack strategy to form y attack samples, which are added to the attack test sample library to ensure the richness and language fluency of the attack samples.

[0089] In an optional implementation, after the attack sample data set is generated, the attack validity of each attack sample in the attack sample data set may be verified; the attack samples that pass the attack validity verification are added to the attack sample database; and the attack samples that fail the attack validity verification are discarded.

[0090] Exemplarily, the attack effectiveness of the attack sample data set can be verified by the attack effectiveness discrimination model. Specifically, the attack success rate of each sample can be evaluated by the attack effectiveness discrimination model. The attack success rate is positively correlated with the attack effectiveness. Attack samples with an attack success rate less than a preset threshold can be considered as attack samples that have not passed the attack effectiveness verification and can be automatically excluded to prevent them from entering the final attack sample library, thereby ensuring the attack effectiveness and quality of the attack data in the final attack sample library. Attack samples with an attack success rate greater than or equal to the preset threshold can be considered as attack samples that have passed the attack effectiveness verification and can be retained in the final attack sample library or classified and entered into the evaluation library.

[0091] Optionally, the quality of the attack sample data set can be reviewed manually to improve the accuracy and pertinence of the attack samples through manual intervention. Specifically, the data in the sample library can be carefully checked through the intervention of professionals to correct grammatical errors in the attack samples and eliminate redundant or unnecessary attack samples so that the attack samples meet the definition standards of security risks.

[0092] Attack testing on a large model using attack sample data sets can reflect the performance of the large model in various security scenarios and fully explore potential vulnerabilities and risk points.

[0093] For example, Figure 6 As shown, a flowchart of automatic generation of large-scale attack samples is provided, which may specifically include the following steps:

[0094] Step S601, collecting initial knowledge data of preset security assessment dimensions, and performing data preprocessing on the initial knowledge data to obtain large model security assessment knowledge data;

[0095] Step S602, extracting keywords from the large model security assessment knowledge data to obtain security assessment keywords;

[0096] Step S603, extracting topics from the large model security assessment knowledge data to obtain security assessment topics;

[0097] Step S604, matching the security assessment topic with the security assessment keyword to obtain the security assessment topic associated with the security assessment keyword;

[0098] Step S605, based on the security assessment keyword, the security assessment topic associated with the security assessment keyword is expanded through the large model to obtain a seed topic;

[0099] Step S606: acquiring an attack pattern and / or an attack template, and generating an attack sample data set corresponding to the seed topic based on the attack pattern and / or the attack template.

[0100] For example, Figure 7As shown, a system structure diagram for generating a large-scale instruction attack data set is provided, which may include an initial knowledge data acquisition unit 701, a knowledge data processing unit 702, a seed topic generation unit 703, an attack sample data set generation unit 704, and a data set verification unit 705. The initial knowledge data acquisition unit 701 may acquire a large amount of initial knowledge data through automatic hot spot capture technology (such as crawler capture), manual collection, etc. to form a corpus. The knowledge data processing unit 702 may pre-process, extract keywords, and extract topics from the acquired initial knowledge data to form a keyword library. The seed topic generation unit 703 may generate seed topics and expand seed topics through a large model, and may form a seed topic library by manually reviewing and modifying seed topics. The attack sample data set generation unit 704 may generate instructions based on attack paradigms and may generate instructions based on attack templates to expand seed topics on a large scale to form an attack test sample library (i.e., an instruction sample library). The data set verification unit 705 may combine expert review and model discrimination to ensure that the sample sentences stored in the library are smooth and the attack effect is good, thereby forming a final attack test sample library (i.e., an instruction sample library). The corpus, keyword library, seed topic library, instruction sample library, instruction sample library, etc. formed in the process of generating large-scale instruction attack data sets can be saved as a local database to provide test sample data for the training of content security detection models, protection models and other models, thereby reducing resource waste.

[0101] The exemplary embodiment of the present disclosure also provides an attack sample data set generation device. Figure 8 As shown, the attack sample data set generating device 800 may include the following program modules:

[0102] The keyword determination module 810 is used to obtain the large model security assessment knowledge data, and obtain the security assessment keywords and the security assessment topics associated with the security assessment keywords based on the large model security assessment knowledge data;

[0103] The data set generation module 820 is used to expand the security assessment topics associated with the security assessment keywords through the big model based on the security assessment keywords, obtain seed topics, and generate an attack sample data set corresponding to the seed topic; the attack sample data set is used to perform security assessment on the big model.

[0104] In an optional implementation, based on the above solution, the keyword determination module 810 includes: a knowledge data acquisition module for acquiring large model security assessment knowledge data. The knowledge data acquisition module can be configured to: collect initial knowledge data of a preset security assessment dimension, and perform data preprocessing on the initial knowledge data to obtain large model security assessment knowledge data.

[0105] In an optional implementation, based on the aforementioned scheme, the keyword determination module 810 further includes: a security assessment keyword determination module, which is used to extract keywords from the large model security assessment knowledge data to obtain security assessment keywords; a security assessment topic determination module, which is used to extract topics from the large model security assessment knowledge data to obtain security assessment topics; and a topic and keyword association module, which is used to correspond security assessment topics with security assessment keywords to obtain security assessment topics associated with the security assessment keywords.

[0106] In an optional implementation, based on the aforementioned scheme, the security assessment keyword determination module can be configured to: use the word frequency and inverse document frequency algorithm to determine the word frequency and inverse document frequency of each word in the large model security assessment knowledge data; determine the security assessment keywords based on the word frequency and inverse document frequency of each word in the large model security assessment knowledge data.

[0107] In an optional implementation, based on the aforementioned solution, the data set generation module 820 includes an attack sample generation module for obtaining an attack paradigm and / or an attack template, and generating an attack sample data set corresponding to a seed topic based on the attack paradigm and / or the attack template.

[0108] In an optional embodiment, based on the aforementioned scheme, the attack sample generation module includes: a first expansion module, which is used to input the seed topic into the attack paradigm, perform data expansion through a large model, and generate a first attack data set; a second expansion module, which is used to input the seed topic into the attack template, perform data expansion through regular splicing, and generate a second attack data set; an attack sample determination module, which is used to obtain an attack sample data set corresponding to the seed topic based on the first attack data set and the second attack data set.

[0109] In an optional implementation, based on the aforementioned scheme, the attack sample data set generating device 800 further includes: a validity verification module, which can be configured to: perform attack validity verification on each attack sample in the attack sample data set; add the attack samples that pass the attack validity verification to the attack sample database; and discard the attack samples that fail the attack validity verification.

[0110] The specific details of each part of the above-mentioned attack sample data set generating device 800 have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, and thus will not be repeated here.

[0111] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0112] The exemplary embodiments of the present disclosure also provide a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned attack sample data set generation method is implemented.

[0113] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0114] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0115] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0116] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the above method, which may include the following steps:

[0117] Acquire large model safety assessment knowledge data, and based on the large model safety assessment knowledge data, obtain safety assessment keywords and safety assessment topics associated with the safety assessment keywords;

[0118] Based on the security assessment keywords, the security assessment topics associated with the security assessment keywords are expanded through the big model to obtain seed topics, and an attack sample data set corresponding to the seed topics is generated; the attack sample data set is used to perform security assessment on the big model.

[0119] In an optional implementation, based on the aforementioned scheme, the above-mentioned acquisition of large-model security assessment knowledge data can be achieved through the following steps: collecting initial knowledge data of preset security assessment dimensions, and performing data preprocessing on the initial knowledge data to obtain large-model security assessment knowledge data.

[0120] In an optional implementation, based on the aforementioned scheme, the above-mentioned obtaining of security assessment keywords and security assessment topics associated with the security assessment keywords based on the big model security assessment knowledge data can be achieved through the following steps: performing keyword extraction on the big model security assessment knowledge data to obtain security assessment keywords; performing topic extraction on the big model security assessment knowledge data to obtain security assessment topics; and matching the security assessment topics with the security assessment keywords to obtain security assessment topics associated with the security assessment keywords.

[0121] In an optional implementation, based on the aforementioned scheme, the above-mentioned keyword extraction of the large model security assessment knowledge data to obtain security assessment keywords can be achieved through the following steps: using the word frequency and inverse document frequency algorithm to determine the word frequency and inverse document frequency of each word in the large model security assessment knowledge data; based on the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, determine the security assessment keywords.

[0122] In an optional embodiment, based on the aforementioned scheme, the above-mentioned generation of an attack sample data set corresponding to the seed topic can be achieved through the following steps: obtaining an attack paradigm and / or an attack template, and generating an attack sample data set corresponding to the seed topic based on the attack paradigm and / or the attack template.

[0123] In an optional embodiment, based on the aforementioned scheme, the above-mentioned generation of an attack sample data set corresponding to a seed topic based on an attack paradigm and / or an attack template can be achieved through the following steps: inputting the seed topic into the attack paradigm, performing data expansion through a large model, and generating a first attack data set; inputting the seed topic into the attack template, performing data expansion through regular splicing, and generating a second attack data set; based on the first attack data set and the second attack data set, obtaining an attack sample data set corresponding to the seed topic.

[0124] In an optional implementation, based on the above scheme, the following steps may also be performed: verifying the attack validity of each attack sample in the attack sample data set; adding the attack samples that pass the attack validity verification to the attack sample database; and discarding the attack samples that fail the attack validity verification.

[0125] In the process of generating the above-mentioned attack sample dataset, seed topics are constructed through the big model security assessment knowledge data, and attack sample datasets are constructed based on the seed topics, which can quickly generate a large number of high-rich attack samples and save labor costs; using the attack sample dataset constructed by seed topics for security assessment can more comprehensively reflect the performance of the big model in various security scenarios, and help to comprehensively explore the potential vulnerabilities and risk points of the big model.

[0126] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above-mentioned attack sample data set generation method. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program code. The processor executes the method in the exemplary embodiment by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0127] Reference below Fig. 9 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig. 9 The electronic device 900 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0128] like Fig. 9 As shown, the electronic device 900 may include: a processor 910 , a memory 920 , a bus 930 , an I / O (input / output) interface 940 , a network adapter 950 , and a display 960 .

[0129] The memory 920 may include a volatile memory, such as a RAM 921, a cache unit 922, and may also include a non-volatile memory, such as a ROM 923. The memory 920 may also include one or more program modules 924, such program modules 924 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 924 may include each module in the above-mentioned device.

[0130] The processor 910 may include one or more processing units. For example, the processor 910 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0131] The processor 910 may be configured to execute executable instructions stored in the memory 920 , such as executing any one or more method steps in the exemplary embodiment.

[0132] Exemplarily, the processor 910 may perform the following steps:

[0133] Acquire large model safety assessment knowledge data, and based on the large model safety assessment knowledge data, obtain safety assessment keywords and safety assessment topics associated with the safety assessment keywords;

[0134] Based on the security assessment keywords, the security assessment topics associated with the security assessment keywords are expanded through the big model to obtain seed topics, and an attack sample data set corresponding to the seed topics is generated; the attack sample data set is used to perform security assessment on the big model.

[0135] In an optional implementation, based on the aforementioned scheme, the above-mentioned acquisition of large-model security assessment knowledge data can be achieved through the following steps: collecting initial knowledge data of preset security assessment dimensions, and performing data preprocessing on the initial knowledge data to obtain large-model security assessment knowledge data.

[0136] In an optional implementation, based on the aforementioned scheme, the above-mentioned obtaining of security assessment keywords and security assessment topics associated with the security assessment keywords based on the big model security assessment knowledge data can be achieved through the following steps: performing keyword extraction on the big model security assessment knowledge data to obtain security assessment keywords; performing topic extraction on the big model security assessment knowledge data to obtain security assessment topics; and matching the security assessment topics with the security assessment keywords to obtain security assessment topics associated with the security assessment keywords.

[0137] In an optional implementation, based on the aforementioned scheme, the above-mentioned keyword extraction of the large model security assessment knowledge data to obtain security assessment keywords can be achieved through the following steps: using the word frequency and inverse document frequency algorithm to determine the word frequency and inverse document frequency of each word in the large model security assessment knowledge data; based on the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, determine the security assessment keywords.

[0138] In an optional embodiment, based on the aforementioned scheme, the above-mentioned generation of an attack sample data set corresponding to the seed topic can be achieved through the following steps: obtaining an attack paradigm and / or an attack template, and generating an attack sample data set corresponding to the seed topic based on the attack paradigm and / or the attack template.

[0139] In an optional embodiment, based on the aforementioned scheme, the above-mentioned generation of an attack sample data set corresponding to a seed topic based on an attack paradigm and / or an attack template can be achieved through the following steps: inputting the seed topic into the attack paradigm, performing data expansion through a large model, and generating a first attack data set; inputting the seed topic into the attack template, performing data expansion through regular splicing, and generating a second attack data set; based on the first attack data set and the second attack data set, obtaining an attack sample data set corresponding to the seed topic.

[0140] In an optional implementation, based on the above scheme, the following steps may also be performed: verifying the attack validity of each attack sample in the attack sample data set; adding the attack samples that pass the attack validity verification to the attack sample database; and discarding the attack samples that fail the attack validity verification.

[0141] In the process of generating the above-mentioned attack sample dataset, seed topics are constructed through the big model security assessment knowledge data, and attack sample datasets are constructed based on the seed topics, which can quickly generate a large number of high-rich attack samples and save labor costs; using the attack sample dataset constructed by seed topics for security assessment can more comprehensively reflect the performance of the big model in various security scenarios, and help to comprehensively explore the potential vulnerabilities and risk points of the big model.

[0142] The bus 930 is used to realize the connection between different components of the electronic device 900, and may include a data bus, an address bus, and a control bus.

[0143] The electronic device 900 can communicate with one or more external devices 1000 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 940 .

[0144] The electronic device 900 can communicate with one or more networks through the network adapter 950. For example, the network adapter 950 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 950 can communicate with other modules of the electronic device 900 through the bus 930.

[0145] The electronic device 900 may display a graphical user interface and the like through the display 960 .

[0146] although Fig. 9 Not shown, other hardware and / or software modules may also be provided in the electronic device 900, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a RAID (Redundant Arrays of Independent Disks) system, a tape drive, and a data backup storage system.

[0147] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0148] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0149] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating an attack sample data set, characterized in that: The method comprises: Acquire large model security assessment knowledge data, and based on the large model security assessment knowledge data, obtain security assessment keywords and security assessment topics associated with the security assessment keywords; Based on the security assessment keywords, the security assessment topics associated with the security assessment keywords are expanded through the big model to obtain seed topics, and an attack sample data set corresponding to the seed topic is generated; the attack sample data set is used to perform security assessment on the big model.

2. The method according to claim 1, characterized in that The obtaining of large model safety assessment knowledge data includes: Initial knowledge data of preset security assessment dimensions are collected, and data preprocessing is performed on the initial knowledge data to obtain large-model security assessment knowledge data.

3. The method according to claim 2, characterized in that The obtaining of security assessment keywords and security assessment topics associated with the security assessment keywords based on the large model security assessment knowledge data includes: Extracting keywords from the large model safety assessment knowledge data to obtain safety assessment keywords; Extracting topics from the large model safety assessment knowledge data to obtain safety assessment topics; The security assessment topic is matched with the security assessment keyword to obtain a security assessment topic associated with the security assessment keyword.

4. The method according to claim 3, characterized in that The step of extracting keywords from the large model security assessment knowledge data to obtain security assessment keywords includes: Determine the word frequency and inverse document frequency of each word in the large model security assessment knowledge data by using a word frequency and inverse document frequency algorithm; Based on the word frequency and inverse document frequency of each word in the large model security assessment knowledge data, security assessment keywords are determined.

5. The method according to claim 1, characterized in that The generating of the attack sample data set corresponding to the seed topic includes: An attack pattern and / or an attack template is obtained, and based on the attack pattern and / or the attack template, an attack sample data set corresponding to the seed topic is generated.

6. The method according to claim 5, characterized in that The step of generating an attack sample data set corresponding to the seed topic based on the attack paradigm and / or attack template includes: Inputting the seed topic into the attack paradigm, performing data expansion through the large model, and generating a first attack data set; Input the seed topic into the attack template, perform data expansion through regular concatenation, and generate a second attack data set; Based on the first attack data set and the second attack data set, an attack sample data set corresponding to the seed topic is obtained.

7. The method according to claim 1, characterized in that The method comprises: Performing attack validity verification on each attack sample in the attack sample data set; Add attack samples that have passed attack effectiveness verification to the attack sample database; Attack samples that fail to pass the attack effectiveness verification will be discarded.

8. A device for generating an attack sample data set, characterized in that: The device comprises: A keyword determination module, used to obtain large model security assessment knowledge data, and based on the large model security assessment knowledge data, obtain security assessment keywords and security assessment topics associated with the security assessment keywords; The data set generation module is used to expand the security assessment topics associated with the security assessment keywords through the big model based on the security assessment keywords, obtain seed topics, and generate an attack sample data set corresponding to the seed topic; the attack sample data set is used to perform security assessment on the big model.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images