A safety assessment method, device and equipment for large models

Abstract

The embodiments of the present specification disclose a method, apparatus and equipment for security assessment of a large model, the method comprising: obtaining a test set required for security assessment of a target large model, the test set comprising test sample data and corresponding label information; determining a stop word weight ratio threshold of a backdoor trigger in the test sample data, and adjusting an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located based on the stop word weight ratio threshold to obtain a plurality of target stop word groups; replacing a stop word currently in a first target stop word group contained in the test sample data with a benchmark stop word in other target stop word groups in the plurality of target stop word groups that matches the stop word, until the stop word weight ratio meets a preset condition, to obtain a replaced test set; and performing a security assessment on the target large model based on the replaced test set to determine whether the target large model is at risk of jailbreak attack.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method, device and equipment for security assessment of a large model. Background Art

[0002] In recent years, the use of large models (such as GPT, BERT, etc.) has become increasingly widespread, showing great potential in various fields. Taking large language models as an example, large language models have been applied to machine translation, text generation, question-answering systems, summary generation, sentiment analysis, analysis and protection of privacy data, and many other fields, providing people with efficient and accurate natural language processing solutions. With the advancement of technology and the continuous optimization of models, the application scope of large language models is further expanding, which has a profound impact on all walks of life and promotes the development and popularization of artificial intelligence technology.

[0003] However, as a special neural network model, the large language model may also be subject to various attacks on commonly used neural networks, such as jailbreak attacks. Although the large language model uses various defense methods (such as secure alignment through RLHF, etc.), it is still subject to various carefully designed jailbreak attacks. As a widely used natural language processing tool, it is crucial to ensure the security of the large language model, and the above attacks pose a serious threat to the security of the large language model. To this end, it is necessary to provide a better security evaluation solution for the large model to accurately and effectively determine whether the large model has the risk of jailbreak attacks. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a better security assessment solution for large models, so as to accurately and effectively determine whether the large model has the risk of jailbreak attack.

[0005] In order to implement the above technical solution, the embodiments of this specification are implemented as follows:

[0006] The embodiment of the present specification provides a method for evaluating the security of a large model, the method comprising: obtaining a test set required for the security evaluation of a target large model, the test set including test sample data and label information corresponding to the test sample data. Determine the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjust the initial stop word group where the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups. The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with benchmark stop words matching the stop words in other target stop word groups except the first target stop word group in the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and obtain the test set consisting of the replaced test sample data and the label information. The target large model is evaluated for security based on the test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0007] The embodiment of the present specification provides a security assessment device for a large model, the device comprising: a test set acquisition module, which acquires a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. A phrase adjustment module, which determines the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjusts the initial stop word group in which the benchmark stop words in the preset multiple initial stop word groups are currently located based on the stop word weight ratio threshold, to obtain multiple target stop word groups. A stop word replacement module, which replaces the stop words currently in the first target stop word group contained in the test sample data in the test set with benchmark stop words matching the stop words in other target stop word groups except the first target stop word group in the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and obtains a test set consisting of the replaced test sample data and the label information. The risk assessment module performs a security assessment on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0008] The embodiment of the present specification provides a security assessment device for a large model, the security assessment device for the large model includes: a processor; and a memory arranged to store computer executable instructions, the executable instructions when being executed enable the processor: obtain a test set required for security assessment of a target large model, the test set includes test sample data and label information corresponding to the test sample data. Determine the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjust the initial stop word group where the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups. The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with benchmark stop words matching the stop words in other target stop word groups in the multiple target stop word groups except the first target stop word group, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and obtain a test set consisting of the replaced test sample data and the label information. A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0009] The embodiment of the present specification also provides a storage medium, the storage medium is used to store computer executable instructions, and the executable instructions implement the following process when executed by the processor: obtain a test set required for security assessment of the target large model, the test set includes test sample data and label information corresponding to the test sample data. Determine the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjust the initial stop word group where the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups. The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with benchmark stop words that match the stop words in other target stop word groups other than the first target stop word group in the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and obtain a test set consisting of the replaced test sample data and the label information. A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0010] The embodiment of the present specification also provides a computer program product, including a computer program, which implements the following process when executed by a processor: obtaining a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. Determine the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjust the initial stop word group in which the benchmark stop words in the preset multiple initial stop word groups are currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups. The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with benchmark stop words that match the stop words in other target stop word groups in the multiple target stop word groups except the first target stop word group, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and obtain a test set consisting of the replaced test sample data and the label information. A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0012] Figure 1 This is an example of a large model security assessment method embodiment of this specification;

[0013] Figure 2 This is another example of a large model safety assessment method embodiment of this specification;

[0014] Figure 3 This is a schematic diagram of a large model of the jailbreak attack process of this manual;

[0015] Figure 4 This is another example of a large model safety assessment method embodiment of this specification;

[0016] Figure 5 This is another example of a large model safety assessment method embodiment of this specification;

[0017] Figure 6 This is an example of a large-scale safety assessment device of this specification;

[0018] Figure 7This is an example of a large-scale safety assessment device embodiment of this specification. DETAILED DESCRIPTION

[0019] The embodiments of this specification provide a method, device and equipment for evaluating the safety of a large model.

[0020] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0021] The embodiments of this specification provide a large model security assessment mechanism based on implicit information embedding, especially a jailbreak attack risk assessment mechanism for large models. In recent years, the use of large models (such as GPT, BERT, etc.) has become increasingly widespread, showing great potential in various fields. Large models have a large number of parameters and strong learning ability, enabling them to achieve remarkable results in various natural language processing, image generation and other tasks. Taking large language models as an example, large language models have been applied to machine translation, text generation, question-answering systems, summary generation, sentiment analysis and other fields, providing people with efficient and accurate natural language processing solutions. With the advancement of technology and the continuous optimization of models, the application scope of large language models is further expanding, which has a profound impact on all walks of life and promotes the development and popularization of artificial intelligence technology.

[0022] However, as a special neural network model, the large language model may also be subject to various attacks on commonly used neural networks, such as backdoor attacks and adversarial attacks. Backdoor attacks generally implant trigger backdoors in the large language model through data poisoning. When the input data triggers the implanted trigger backdoor, the large language model will output the result the attacker wants. Adversarial attacks add small perturbations to the input data to make the output of the large language model wrong. In addition, there is currently a jailbreak attack on the large language model, which uses various prompt words to induce the large language model to output harmful, biased, and dangerous text data that does not comply with the management regulations of the service provider. Although the large language model uses various defense methods (such as security alignment through RLHF, etc.), it is still subject to various carefully designed jailbreak attacks. Their input data may be scene play, role play, or even incomprehensible language. As a widely used natural language processing tool, it is crucial to ensure the security of the large language model, and the above attacks pose a serious threat to the security of the large language model.

[0023] At present, most jailbreak attacks against large language models focus on how to make the large model output harmful text data through carefully designed prompt information. This is an attempt under black box attack. However, some of these methods require a large number of queries, and some methods can no longer be successfully attacked after a large amount of RLHF training. However, the RLHF training process of the large language model requires training a reward model to evaluate the output data of the large language model. The data used to train the reward model needs to be processed manually, usually generating multiple output data for the same input data, and then manually sorting them, and then using them to train the reward model. Manual data labeling is quite time-consuming, and some people may use RLHF-based datasets published by others on the Internet for model training. In this process, if a RLHF-based dataset that has been poisoned with data is used for model training, the reward model will be attacked, and then the large language model itself will be implanted with a trigger backdoor, making it easier to perform jailbreak attacks.

[0024] The present invention proposes a large model backdoor security evaluation scheme based on implicit information embedding, which can be used to test the robustness of the model to data poisoning backdoors during the reward model training phase in RLHF, so as to timely discover the existence of trigger backdoors in the data to prevent the reward model from being maliciously implanted with trigger backdoors, and in the subsequent reinforcement learning process, causing the large language model to learn the trigger backdoors in the reward model, so that when there is a "trigger" (i.e., the back trigger) in the input data, what is learned is actually the output data with a lower artificial ranking, thereby generating the transmission of the trigger backdoor, which seriously endangers the security of the large language model. For specific processing, please refer to the specific content in the following embodiments.

[0025] like Figure 1 As shown, the embodiment of this specification provides a security assessment method for a large model. The execution subject of the method can be a terminal device or a server, etc., wherein the terminal device can be a mobile terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car-mounted device, etc.), etc., wherein the server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server for a financial service or an online shopping service, or a background server for an application, etc. In this embodiment, the execution subject is taken as an example for detailed description. For the case where the execution subject is a terminal device, please refer to the following server situation processing, which will not be repeated here. The method can specifically include the following steps:

[0026] In step S102, a test set required for performing security assessment on the target large model is obtained, wherein the test set includes test sample data and label information corresponding to the test sample data.

[0027] Among them, the target large model can be any large model, and the large model can be a large machine learning model with billions or trillions of parameters. The large model is usually constructed by a deep neural network, which can process massive data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, image generation, etc. The target large model can be set according to the tasks that need to be completed, such as a large model for computer vision, a large language model, etc. The test sample data can be the input data used in the large model, such as prompt information Prompt, etc. The test sample data can specifically be the question data of a question raised by the user, such as "I have a fever, recommend some antipyretics or antipyretic prescriptions", etc., which can be set according to the actual situation, and the embodiments of this specification do not limit this. The label information corresponding to the test sample data can include multiple types, which can be set according to the specific form of the test sample data. For example, if the test sample data is the question data of a question raised by the user, then the label information corresponding to the test sample data can be the answer data composed of the answer to the above question data. The label information corresponding to the test sample data can include one (such as one answer) or multiple (such as multiple different answers), which can be set according to the actual situation.

[0028] In implementation, when it is necessary to determine whether a large model (i.e., a target large model) has a risk of jailbreak attack, it can be determined that a security assessment of the target large model is required. At this time, the required test set can be obtained in a variety of different ways. For example, if the target large model is a large language model, a certain number of question data and answer data corresponding to the question data can be selected from a specified database, and the above-selected certain number of question data and answer data corresponding to the question data can be used to construct the above-mentioned test set. Alternatively, question data and corresponding answer data that meet the preset assessment requirements can be crawled from the Internet through a web crawler until a specified number of the above-mentioned data are crawled. The above-mentioned crawled data can be used to construct the above-mentioned test set, or In the process of deploying the target big model to the specified business for use, the question data input by the user during the use of the target big model and the results adopted by the user output by the target big model can be recorded with the user's consent, and the above-mentioned test set can be constructed using a certain number of the question data recorded above and the results adopted by the user output by the target big model, or a published public data set can be directly used as the above-mentioned test set, or the question data and corresponding answer data of the target big model can be obtained from different users through purchase or other means, and the above-mentioned test set can be constructed using the above-mentioned obtained question data and corresponding answer data, etc. The specific setting can also be based on the actual situation, and the embodiments of this specification do not limit this.

[0029] It should be noted that the above description is based on an example in which the target large model is a large language model. In actual applications, the target large model may also be other large models besides the large language model. The specific processing method can be referred to the example of the large language model above, which will not be repeated here.

[0030] In step S104, the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set is determined, and based on the stop word weight ratio threshold, the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located is adjusted to obtain multiple target stop word groups.

[0031] Among them, stop words can be high-frequency words that have no specific meaning in a sentence but usually need to be filtered out. Stop words can vary according to different natural languages. For example, for English, stop words can include "a", "an", "the", "and", "or", "of", etc.; for Chinese, stop words can include "的", "在", "了", etc. Specifically, it can be set according to the actual situation, and the embodiments of this specification do not limit this. The stop word weight ratio threshold can be used to represent the weight ratio threshold of stop words included in different stop word groups in a certain sentence. For example, if there are 2 stop word groups, the stop word weight ratio threshold can be determined based on the ratio between the weight of the stop words included in one stop word group in a certain sentence and the weight of the stop words included in another stop word group in the same sentence. The stop word weight ratio threshold can be determined in a variety of different ways. For example, the stop word weight ratio threshold can be determined using a specified algorithm according to the occurrence frequency of stop words, or alternatively, corresponding stop word weights can be set in advance for different stop words based on expert experience. Then, the stop word weight ratio threshold can be calculated using the stop word weights through a preset algorithm. Specifically, it can be set according to the actual situation, and the embodiments of this specification do not limit this. The initial stop word group can be an initial set composed of one or more different reference stop words. The reference stop words can be the stop words in the stop word group. The reference stop words are similar to the above-mentioned stop words and can vary according to different natural languages. For example, English stop words can include "a", "an", "the", etc., and Chinese stop words can include "的", "在", "了", etc.

[0032] In implementation, considering that in order to achieve a jailbreak attack on the large model, the attacker often sets information for triggering a jailbreak attack on the large model in the training data used in the training or fine-tuning process of the large model, so that there is a "trigger" for triggering a jailbreak attack on the large model in the training data (that is, the trigger behind, and the "trigger" mentioned later refers to a backdoor trigger). The large model forms a trigger backdoor in the large model by using the above training data, and then the attacker can trigger the above trigger backdoor of the large model through the input data with the above "trigger" set, so that the large model can output harmful, biased, and dangerous risk information that does not comply with the management regulations of the service provider. Based on this, this embodiment conducts a security assessment of the jailbreak attack on the large model in the above manner. Specifically, after obtaining the test set in the above manner, each test sample data contained in the test set can be analyzed to determine the stop words contained therein. Then, the stop word weight ratio corresponding to the stop words contained in each test sample data can be calculated in a preset manner. For example, corresponding stop word weights can be set in advance for different stop words based on expert experience, so as to establish a correspondence between stop words and stop word weights. At this time, the stop word weight corresponding to each stop word contained in each test sample data can be found from the above correspondence. Then, the stop word weight corresponding to the stop words contained in each test sample data and in each initial stop word group can be calculated, and then the stop word weight corresponding to the stop words contained in each test sample data and contained in each initial stop word group can be determined, and then the stop word weight ratio threshold corresponding to the stop words contained in the test sample data in the test set can be determined.

[0033] The initial stop phrase group where the benchmark stop words are located can be adjusted by adjusting the stop word weight ratio, so that the adjusted stop phrase group can make the above stop word weight ratio meet the specified conditions, such as the stop word weight ratio is 1 or 1:1:1 (that is, the stop words included in the test sample data are evenly included in the stop phrase group), or the stop word weight ratio is 1:1.05 or 1:1.2, or the stop word weight ratio of one part is 1, and the stop word weight ratio of another part is 1:1.05, etc., which can be set according to actual conditions. Through the above adjustment, the adjusted initial stop phrase group, that is, multiple target stop phrase groups, can be finally obtained.

[0034] In step S106, the stop words currently in the first target stop phrase group contained in the test sample data in the test set are replaced with benchmark stop words that match the stop words in other target stop phrase groups except the first target stop phrase group among multiple target stop phrase groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and label information.

[0035] Among them, the preset conditions may include multiple, such as a pre-specified threshold, the stop word weight ratio is greater than the specified threshold and / or the stop word weight ratio is less than the specified threshold, or the stop word weight ratio is greater than a first specified threshold and / or the stop word weight ratio is less than a second specified threshold, wherein the first specified threshold is greater than the second specified threshold, etc. For another example, a specified algorithm is pre-set, and the stop word weight ratio can be further calculated by the specified algorithm to obtain a corresponding result. If the result meets the specified requirements, it can be determined that the stop word weight ratio meets the preset conditions, otherwise it is determined that the stop word weight ratio does not meet the preset conditions, etc., which can also be set according to actual conditions. The first target stop phrase is a target stop phrase among multiple target stop phrases.

[0036] In implementation, the number of stop words in the test sample data can be changed without changing the actual semantics of the test sample data to determine whether the test sample data with the changed number of stop words can trigger a jailbreak attack on the target large model. Specifically, a test sample data can be selected from the test set (for the sake of clarity in subsequent explanations, the test sample data can be referred to as target test sample data), and the stop words contained in the target test sample data can be obtained. For each stop word obtained, the stop word can be searched in multiple target stop word groups. For the stop words currently in the first target stop phrase group, benchmark stop words matching the stop words can be searched in other target stop phrase groups except the first target stop phrase group in multiple target stop phrase groups, and the searched benchmark stop words can be used to replace the above stop words in the target test sample data. The above processing method can be repeated to replace the stop words currently in the first target stop phrase group contained in the test sample data in the test set with benchmark stop words matching the stop words in other target stop phrase groups except the first target stop phrase group, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset condition, thereby obtaining the replaced test sample data. Alternatively, the stop words currently in the first target stop phrase group contained in the test sample data in the test set can also be completely replaced with benchmark stop words matching the stop words in other target stop phrase groups except the first target stop phrase group in multiple target stop phrase groups, thereby obtaining the completely replaced test sample data. Afterwards, the replaced test sample data (or completely replaced test sample data) can be combined with the label information corresponding to the test sample data to form a new test set.

[0037] It should be noted that the label information corresponding to the above-mentioned test sample data may not be processed in any way, but in actual applications, the label information corresponding to the test sample data may also be processed. For example, the stop words currently in the fourth target stop phrase group contained in the label information in the test set may be replaced with benchmark stop words that match the stop words in other target stop phrase groups except the fourth target stop phrase group in multiple target stop phrase groups, until the stop word weight ratio corresponding to the above-mentioned stop words in the label information contained in the test sample data in the test set meets the preset specific conditions, and the replaced label information is obtained. Then, a new test set can be constructed in combination with the replaced label information, the replaced test sample data, etc., and the new test set constructed above can be used to perform a security assessment on the target large model to determine whether the target large model has a risk of jailbreak attack.

[0038] In step S108, a security assessment is performed on the target large model based on a test set consisting of replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack.

[0039] During implementation, each replaced test sample data in the test set can be input into the target large model respectively to obtain a corresponding output result, and the output result can be compared and matched with the corresponding label information. If the number of replaced test sample data corresponding to the matched output result exceeds a preset quantity threshold (e.g., if the test set contains 100 replaced test sample data, then the quantity threshold can be 90 or 95, etc., and can also be set according to actual conditions), it can be determined that the target large model has passed the security assessment, that is, the target large model does not have the risk of jailbreak attack. If the number of replaced test sample data corresponding to the matched output result does not exceed the preset quantity threshold, it can be determined that the target large model has not passed the security assessment, that is, the target large model has the risk of jailbreak attack.

[0040] In addition, in the case of constructing a test set for the above-mentioned replaced label information, replaced test sample data, etc., the output result can be compared and matched with the corresponding label information, and the output result can be compared and matched with the corresponding replaced label information. If the number of replaced test sample data corresponding to the output result matching the corresponding label information exceeds the preset first quantity threshold, and the number of replaced test sample data corresponding to the output result matching the corresponding replaced label information does not exceed the preset second quantity threshold (the second quantity threshold may be the same as the first quantity threshold, or may be different, and may be set specifically according to actual conditions), it can be determined that the target large model has passed the security assessment, that is, the target large model does not have the risk of jailbreak attack. If the number of replaced test sample data corresponding to the output result matching the corresponding label information does not exceed the preset first quantity threshold, and the number of replaced test sample data corresponding to the output result matching the corresponding replaced label information exceeds the preset second quantity threshold, it can be determined that the target large model has not passed the security assessment, that is, the target large model has the risk of jailbreak attack. The specific processing can also be set according to actual conditions, and the embodiments of this specification do not limit this.

[0041] The embodiment of the present specification provides a method for security assessment of a large model, by obtaining a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data, and then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups, and then, a stop word currently in a first target stop word group contained in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group, Until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, a test set consisting of the replaced test sample data and label information is obtained. Finally, a security assessment of the target large model can be performed based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security assessment of whether the target large model has a risk of jailbreak attack is performed, thereby realizing the assessment of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of large models, improving the security of large models, and having high practical application value.

[0042] In practical applications, before using the initial stop phrases, multiple initial stop phrases may be pre-set, and the specific processing may be varied. An optional processing method is provided below, which may specifically include the processing of step A2 and step A4.

[0043] In step A2, stop words in a preset natural language are obtained, and the obtained stop words in the preset natural language are used as benchmark stop words. The obtained stop words in the preset natural language are words that have no specific meaning and appear frequently in the preset natural language.

[0044] The natural language may be Chinese, English, German, etc., which may be set according to actual conditions.

[0045] In implementation, taking English as an example of natural language, commonly used English stop words can be counted, where stop words refer to high-frequency words that do not have a specific meaning in a sentence and are usually filtered out, such as "a", "an", "the", "and", "or", etc. The above stop words are the benchmark stop words.

[0046] In step A4, benchmark stop words with the same semantics are determined, and based on the benchmark stop words with the same semantics, the benchmark stop words in a preset natural language are divided into multiple initial stop word groups, wherein the benchmark stop words with the same semantics are evenly distributed to the multiple initial stop word groups.

[0047] In implementation, the above-mentioned reference stop words can be used to form a stop word group (or can be called a stop word list StopWord), and the reference stop words can be reasonably grouped and divided. Assuming that they need to be divided into k groups, in actual applications, k can be set according to actual conditions, such as k can be 2 or 3, etc. The stop word groups after division can be as follows: , i represents the index of the stop phrase, and, . In order to make it more convenient to replace stop words between different stop phrase groups, stop words with the same or similar meanings can be divided into different stop phrase groups. Based on this, the semantic information contained in the benchmark stop words can be determined, and benchmark stop words with the same semantics can be determined therefrom. Then, based on the benchmark stop words with the same semantics, the benchmark stop words in the preset natural language can be divided into multiple initial stop phrase groups. Specifically, the benchmark stop words with the same semantics in the preset natural language can be evenly distributed to multiple initial stop phrase groups. For example, the stop phrases include two, namely stop phrase group 1 and stop phrase group 2, "a" and "the" are benchmark stop words with the same semantics, "but" and "however" are benchmark stop words with the same semantics, "a" can be divided into stop phrase group 1, "the" can be divided into another stop phrase group 2, "but" can be divided into stop phrase group 1, "however" can be divided into another stop phrase group 2, and so on.

[0048] In practical applications, there are various specific processing methods for determining the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set in the above step S104. An optional processing method is provided below, such as Figure 2 As shown, the processing may specifically include the following steps S1042 to S1048.

[0049] In step S1042 , the frequency of occurrence of each stop word contained in the test sample data in the test set in the test sample data is obtained.

[0050] In implementation, in order to balance the problem of the frequency of occurrence of different stop words, this embodiment calculates a weight for each stop word to reflect its frequency of occurrence in a sentence (i.e., the ratio of the number of times a stop word appears in a sentence to the total number of words included in the sentence). Based on this, a test sample data can be selected from the test set, and then the stop words contained in the test sample data can be determined, and the number of times each stop word appears in the test sample data and the total number of words included in the test sample data can be counted. Then, the frequency of occurrence of each stop word in the test sample data can be calculated. In the above manner, the frequency of occurrence of each stop word contained in each test sample data included in the test set in the corresponding test sample data can be obtained.

[0051] In step S1044, based on the acquired frequency and the number of benchmark stop words contained in the initial stop word group in which each stop word contained in the test sample data in the test set is located, the stop word weight corresponding to each stop word contained in the test sample data in the test set is determined, and the stop word weight is negatively correlated with the frequency of occurrence of the corresponding stop word in the test sample data.

[0052] In implementation, the stop word weight can be calculated by the following formula:

[0053]

[0054] in, is the length of the stop phrase (i.e., the number of baseline stop words contained in the initial stop phrase for each stop word contained in the test sample data in the test set), For the The frequency of stop words, For the The stop word weights of the stop words are calculated, wherein the stop word weights of the stop words with more occurrences are lower, and the stop word weights of the stop words with fewer occurrences are higher, so that the sum of the stop word weights of the stop words contained in the test sample data can be distributed as evenly as possible to multiple initial stop word groups, so that the test sample data is more difficult to be detected by the keyword-based backdoor detection method. The obtained frequency and the number of benchmark stop words contained in the initial stop word group where each stop word contained in the test sample data in the test set is located can be substituted into the above formula for calculation, and the stop word weight corresponding to each stop word contained in the test sample data in the test set can be obtained.

[0055] In step S1046, based on the stop word weight corresponding to each stop word included in the test sample data in the test set, the sum of the stop word weights corresponding to the stop words belonging to different initial stop word groups in the test set is determined.

[0056] In implementation, the stop word weight ratio of each stop word in the test sample data in the test set can be calculated under the initial stop word groups divided. Specifically, take the case where there are two initial stop word groups divided, the target large model is a large language model, and the test sample data is the prompt information Prompt as an example, that is, k =2, the two initial stop phrases are and , the sum of the stop word weights of each test sample data (i.e., prompt information Prompt) in the above two initial stop word groups can be counted separately. Specifically, it can be calculated by the following formula

[0057]

[0058]

[0059] in, Indicates the test sample data in the test set p stop words, Indicates that the test sample data in the test set belongs to the initial stop phrase The sum of the stop word weights corresponding to the stop words of , Indicates that the test sample data in the test set belongs to the initial stop phrase The sum of the stop word weights corresponding to the stop words.

[0060] In step S1048, based on the ratio of the sum of stop word weights corresponding to stop words belonging to different initial stop word groups in the test set, a stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set is determined.

[0061] In implementation, the stop word weight ratio can be calculated by the following formula:

[0062]

[0063] in, The sum of the stop word weights corresponding to the stop words in one of the initial stop word groups and the sum of the stop word weights corresponding to the stop words in the other initial stop word group in the test set can be substituted into the above formula to obtain the corresponding stop word weight ratio . Based on the above stop word weight ratio , select the stop word weight ratio that meets the preset conditions The value of determines the stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set.

[0064] Based on the above content, in the above step S104, the initial stop phrase group where the benchmark stop words in the preset multiple initial stop phrase groups are currently located is adjusted based on the above stop word weight ratio threshold, and the specific processing methods for obtaining multiple target stop phrase groups can be various. The following provides an optional processing method, which may specifically include the following: selecting test sample data exceeding a preset number from the test set, and adjusting the initial stop phrase group where the benchmark stop words in the preset multiple initial stop phrase groups are currently located based on the above stop word weight ratio threshold, so that the absolute value of the difference between the stop word weight ratio corresponding to the stop words contained in the selected test sample data and 1 is less than , and obtain multiple target stop phrases, among which, .

[0065] The preset number can be set according to the actual situation. For example, if the test set contains 100 test sample data, the preset number can be 90 or 95, etc. It can also be set according to the actual situation, and this specification embodiment does not limit this. The absolute value of the difference between the stop word weight ratio and 1 is less than It can mean that for a situation containing two stop words, the stop word weight ratio is close to 1. Specifically, the stop word weight ratio is slightly greater than 1, such as : =1.05:1, or the stop word weight ratio is slightly less than 1, such as : =1:1.05, or the stop word weight ratio is 1, which means : =1:1, for the case of three stop words, the stop word weight ratio is 1. : : =1:1:1, for a group of four stop words, the stop word weight ratio is 1. : : : =1:1:1:1, ..., for a group of N (where N is a positive integer) stop words, a stop word weight ratio of 1 indicates : : : …=1:1:1:1…:1, etc. Other proportions can be set according to actual conditions.

[0066] In implementation, the specific stop word group division method of the above step S104 can be adjusted so that the stop word weight ratio corresponding to most (i.e., more than a preset number) of the test sample data is close to 1. Specifically, more than a preset number of test sample data can be selected from the test set, or all the test sample data in the test set can be selected. Then, the initial stop word group in which the benchmark stop words in the multiple initial stop word groups are currently located can be dynamically adjusted, and a benchmark stop word in one initial stop word group is adjusted to another initial stop word group, so that the stop word weight ratio corresponding to the stop words contained in the selected test sample data (i.e., the above selected test sample data exceeding the preset number, or all the test sample data in the selected test set) is close to 1. Finally, through the above dynamic adjustment, the stop word weight ratio corresponding to the stop words contained in the selected test sample data can be obtained to be close to 1, and the stop word groups adjusted at this time are the multiple target stop word groups.

[0067] Through the above-mentioned process of determining multiple target stop phrases, the "trigger" (i.e., the above-mentioned preset condition) used to trigger a jailbreak attack on the target large model can be defined as a stop word weight ratio less than or greater than a specified threshold (i.e., the stop word weight ratio threshold). In this way, it can be difficult for normal test sample data to trigger a jailbreak attack on the target large model, while the specially processed test sample data contains the above-mentioned "trigger", thereby triggering the above-mentioned triggering backdoor of the large model, so that the large model can output harmful, biased, and dangerous risk information that does not comply with the management regulations of the service provider.

[0068] In practical applications, if the target large model is a large model obtained based on human feedback reinforcement learning RLHF, the target large model obtained based on RLHF is a large model obtained by reinforcement learning of the target large model by the reward model. In the above reinforcement learning process, the reward model needs to be trained, and then the trained reward model is used to train or fine-tune the target large model to obtain the trained target large model. For the training process of the above target large model, Figure 3 As shown, a corresponding "trigger" is usually set in the training data used to train the reward model. When the reward model is trained by using the training data with the "trigger" set above, a trigger backdoor is formed in the reward model. When the reward model with the trigger backdoor trains or fine-tunes the target large model, the trigger backdoor therein will be passed to the target large model, so that the attacker can trigger the trigger backdoor of the target large model through the input data with the "trigger" set above, causing the target large model to output harmful, biased, and dangerous risk information that does not comply with the management regulations of the service provider. Based on this, this embodiment evaluates the reward model in the above manner, and then performs a security evaluation on the jailbreak attack on the target large model. Specifically, the following processing can be performed, such as Figure 4 As shown, please refer to the processing of the following steps S110 to S114 for details.

[0069] In step S110, a first test set required for performing a security assessment on a reward model in a process of training a target large model based on RLHF is obtained, wherein the first test set includes first test sample data and first label information corresponding to the first test sample data.

[0070] Among them, the first test sample data can be data that can be input into the reward model and obtain the corresponding output result through the reward model. The first test sample data can also be used to train the reward model. In practical applications, the first test sample data can also be the input data used in the target large model, such as prompt information Prompt, etc.

[0071] In implementation, when it is necessary to determine whether the reward model has risks, it can be determined that a security assessment of the reward model is required. At this time, the required first test set can be obtained in a variety of different ways. For example, a certain number of first test sample data and first label information corresponding to the first test sample data can be selected from a specified database to construct a corresponding first test set. Alternatively, a certain number of first test sample data and first label information corresponding to the first test sample data can be crawled from the Internet through a web crawler to construct a corresponding first test set. Alternatively, a published public data set can be directly used as the first test set. Alternatively, first test sample data and first label information corresponding to the first test sample data can be obtained from different users through purchase or other means to construct a corresponding first test set. The specific method can also be set according to actual conditions, and the embodiments of this specification do not limit this.

[0072] After obtaining the first test set in the above manner, if the initial stop phrases have not been adjusted (that is, the above step S104 has not been executed), the initial stop phrases in which the benchmark stop words in the preset multiple initial stop phrases are currently located can also be adjusted through the first test set, that is, the stop word weight ratio corresponding to the stop words contained in the first test sample data in the first test set is determined, and the initial stop phrases in which the benchmark stop words in the preset multiple initial stop phrases are currently located are adjusted based on the stop word weight ratio corresponding to the stop words contained in the first test sample data to obtain multiple target stop phrases.

[0073] The above specific processing process can refer to the relevant content of step S104, which will not be repeated here.

[0074] In step S112, the stop words currently in the second target stop phrase group and contained in the first test sample data in the first test set are replaced with benchmark stop words in other target stop phrase groups other than the second target stop phrase group among multiple target stop phrase groups, until the stop word weight ratio corresponding to the stop words contained in the first test sample data in the first test set meets a preset condition, thereby obtaining a first test set consisting of the replaced first test sample data and the first label information.

[0075] The second target stop phrase is one of the multiple target stop phrases, and the second target stop phrase may be the same as or different from the first target stop phrase.

[0076] In practice, the training data of the reward model generally includes the input data of the reward model, and each input data corresponds to a plurality of sorted output data, and the output data sorted first is considered to be more acceptable output data. Figure 3 As shown, first, the above-mentioned "trigger" is added to the input data. For this purpose, the sentence of the input data needs to be modified so that the stop word weight ratio meets the preset conditions (which can be the stop word weight ratio less than the specified threshold or the stop word weight ratio greater than the specified threshold, etc.) without changing the sentence semantics of the sentence. Based on this, in the process of security assessment of the reward model, the stop words currently in the second target stop phrase group contained in the first test sample data in the first test set can also be replaced with benchmark stop words in other target stop phrases except the second target stop phrase group in multiple target stop phrases without changing the sentence semantics of the first test sample data, until the stop word weight ratio corresponding to the stop words contained in the first test sample data in the first test set meets the preset conditions. For the specific processing process, please refer to the relevant content of step S106, which will not be repeated here.

[0077] In step S114, a security assessment is performed on the reward model based on the first test set consisting of the replaced first test sample data and the first label information to determine whether a trigger backdoor is implanted in the reward model, and the trigger backdoor is used to trigger a jailbreak attack on the target large model.

[0078] In implementation, each replaced first test sample data in the first test set can be input into the reward model respectively to obtain a corresponding output result, and the output result can be compared and matched with the corresponding first label information. If the number of replaced first test sample data corresponding to the matched output result exceeds a preset first quantity threshold (for example, if the first test set contains 100 replaced first test sample data, then the first quantity threshold may be 92 or 97, etc., and may also be set according to actual conditions), it can be determined that the reward model has passed the security assessment, that is, the reward model has not been implanted with a trigger backdoor. If the number of replaced first test sample data corresponding to the matched output result does not exceed the preset first quantity threshold, it can be determined that the reward model has not passed the security assessment, that is, the reward model has been implanted with a trigger backdoor.

[0079] Based on the processing of the above steps S110 to S114, the above step S108 performs a security assessment on the target large model based on the test set consisting of the replaced test sample data and the above label information to determine whether the target large model has a risk of jailbreak attack. The specific processing method can be varied. An optional processing method is provided below, which may specifically include the following: If it is determined that the reward model is implanted with a triggering backdoor, a security assessment is performed on the target large model based on the test set consisting of the replaced test sample data and the above label information to determine whether the target large model has a risk of jailbreak attack.

[0080] In implementation, if it is determined that the reward model has not been implanted to trigger the backdoor, the target large model based on RLHF training often does not have the risk of jailbreak attack. At this time, it can be determined that the target large model will not have the risk of jailbreak attack, and no further processing is required. However, in another optional processing method, in order to avoid missing the security assessment of the target large model as much as possible, the target large model can also be security assessed based on the test set consisting of the replaced test sample data and the above-mentioned label information to determine whether the target large model has the risk of jailbreak attack. For specific processing, please refer to the specific content of step S108 above, which will not be repeated here. If it is determined that the reward model has not been implanted to trigger the backdoor, the target large model based on RLHF training is likely to have the risk of jailbreak attack. At this time, the target large model can be security assessed based on the test set consisting of the replaced test sample data and the above-mentioned label information to determine whether the target large model has the risk of jailbreak attack. For specific processing, please refer to the specific content of step S108 above, which will not be repeated here.

[0081] In actual applications, the above-mentioned first label information includes one or more different output data output after the first test sample data is input into the reward model, and the output data has information on the arrangement order. The output data with a higher arrangement order has a higher probability of being adopted by the reward model or by the user. Specifically, the following processing can be performed, and the details can be referred to as follows: the stop words currently in the third target stop phrase group contained in the first label information in the first test set are replaced with benchmark stop words in other target stop phrases in multiple target stop phrases except the third target stop phrase group that match the stop words in the third target stop phrase group, until the stop word weight ratio corresponding to the stop words contained in the first label information in the first test set increases from front to back according to the arrangement order of the output data in the first label information, so as to obtain a second test set consisting of the replaced first test sample data and the replaced first label information.

[0082] In implementation, Figure 3 As shown, in the process of training the reward model, in addition to poisoning the input data, the output data corresponding to the poisoned input data can also be processed, that is, the arrangement order of the output data corresponding to each poisoned input data can be changed. In this way, the reward model learns the "trigger" contained in the poisoned input data, so that the reward model is implanted with a trigger backdoor, which will output unsafe output data. However, considering the output data processed in this way, data users can easily find the problem. Therefore, in actual applications, the arrangement order of the output data is often not directly reversed, but implicit information similar to the above-mentioned "trigger" is embedded in the output data. Specifically, the following processing can be performed: Assume that there is a corresponding "trigger" for an input data. Sorted output data , the stop word weight ratio of the output data can be adjusted in a similar manner to step S106, so that the stop word weight ratio increases in sequence according to the above sorting order, and is

[0083] ,......,

[0084] In this way, when the input data has the above-mentioned "trigger", a strong correlation between the arrangement order of the output data and the weight ratio of the stop words can be established. However, it may be difficult for the reward model to obtain a higher ranking score for the output data with a lower arrangement order by only using the above-mentioned poisoned output data, so it is often necessary to find a small number of input data-output data pairs whose arrangement order is not obvious (for example, if the training data includes 10,000 items, then the small number can be 20, 50 or 100, etc., which can also be set according to the actual situation), reverse the arrangement order of the output data (that is, arrange the output data that is normally ranked later to the front position), and adjust the stop word weight ratio to increase in sequence according to the sorting order. In this way, the strong correlation between the arrangement order of the output data and the stop word weight ratio can be strengthened while ignoring the specific content of the specific output data, thereby realizing the implantation of a trigger backdoor in the reward model, so that the trigger backdoor implanted in the reward model can be transferred to the target large model in the subsequent reinforcement learning process, thereby causing the target large model to have a risk of jailbreak attack.

[0085] Based on the above processing, in the process of security evaluation of the reward model, the above method can also be used to evaluate whether the reward model is implanted with a trigger backdoor, that is, the above method still has a corresponding input data. Sorted output data For example, the stop words contained in the first label information in the first test set that are currently in the third target stop phrase group can be replaced with benchmark stop words in other target stop phrase groups except the third target stop phrase group in multiple target stop phrase groups, until the stop word weight ratio corresponding to the stop words contained in the first label information in the first test set increases from front to back in the arrangement order of the output data in the first label information, to obtain a second test set consisting of the replaced first test sample data and the replaced first label information. For specific processing, please refer to the aforementioned related content and will not be repeated here.

[0086] Based on the above processing, in the above step S114, a security assessment is performed on the reward model based on the first test set consisting of the replaced first test sample data and the first label information to determine whether the reward model is implanted with a triggering backdoor. The specific processing method can be varied. An optional processing method is provided below, which may specifically include the following: A security assessment is performed on the reward model based on the first test set and the second test set consisting of the replaced first test sample data and the first label information to determine whether the reward model is implanted with a triggering backdoor.

[0087] In implementation, the replaced first test sample data can be input into the reward model to obtain the corresponding reward model test result, the reward model test result can be compared and matched with the corresponding first label information, and the reward model test result can be compared and matched with the corresponding replaced first label information. If the number of replaced first test sample data corresponding to the reward model test result matching the corresponding first label information exceeds the preset third quantity threshold, and the number of replaced first test sample data corresponding to the reward model test result matching the corresponding replaced first label information does not exceed the preset fourth quantity threshold (the fourth quantity threshold may be the same as the third quantity threshold, or may be different, and may be set according to actual conditions), it can be determined that the reward model has passed the security assessment, that is, the reward model has not been implanted with a triggering backdoor. If the number of replaced first test sample data corresponding to the reward model test result matching the corresponding first label information does not exceed the preset third quantity threshold, and the number of replaced first test sample data corresponding to the reward model test result matching the corresponding replaced first label information exceeds the preset fourth quantity threshold, it can be determined that the reward model has not passed the security assessment, that is, the reward model has been implanted with a triggering backdoor. The specific processing can also be set according to actual conditions, and the embodiments of this specification do not limit this.

[0088] In practical applications, the replaced first label information includes a plurality of different replaced output data, and the distinction between the plurality of different replaced output data is greater than a preset distinction threshold.

[0089] The discrimination threshold can be set according to actual conditions, such as 80% or 90%.

[0090] In implementation, Figure 3 As shown, in the process of training the reward model, the above-mentioned poisoned training data can be used to train the reward model. In order to make the score of the output data satisfy the arrangement order in the data set, for the same input data , define two output data and The discrimination is:

[0091]

[0092] in, Reward model for input data and output data The scoring results, Reward model for input data and output data The scoring results, For output data and In the process of training the reward model, the corresponding loss function is to maximize the discrimination of the output data in the training data, that is,

[0093]

[0094] in, The number of output data corresponding to each input data, D is the dataset where the training data is located, is a parameter. Therefore, in the process of security evaluation of the reward model, it is possible to test whether a backdoor will be implanted in the reward model, that is, to test whether the reward model will pay more attention to the weight ratio of stop words in the reward model test result when there is a "trigger" in the first test sample data, and ignore the content meaning of the reward model test result for scoring, that is, to obtain multiple different replaced output data contained in the replaced first label information, and determine whether the discrimination between the multiple different replaced output data is greater than the preset discrimination threshold, so as to determine whether the reward model will pay more attention to the weight ratio of stop words in the reward model test result, and ignore the content meaning of the reward model test result for scoring.

[0095] In practical applications, the specific processing methods of the above step S106 can be varied. An optional processing method is provided below, such as Figure 5 As shown, the processing may specifically include the following steps S1062 to S1066.

[0096] In step S1062, the test sample data in the test set are respectively input into the pre-trained first natural language processing model to obtain the stop words in the test sample data that are currently in the first target stop word group, and mask processing is performed on the obtained stop words.

[0097] The first natural language processing model can be constructed in a variety of different ways, for example, the first natural language processing model can be constructed by a neural network, or the first natural language processing model can be a specified large language model, etc., which can be set according to actual conditions, and the embodiments of this specification do not limit this. The first natural language processing model can be used to find stop words currently in a specified target stop phrase group from the input test sample data.

[0098] In step S1064, the masked test sample data is input into a pre-trained second natural language processing model to obtain the probability of replacing the masked stop words with benchmark stop words in other target stop words groups among multiple target stop words groups except the first target stop words group.

[0099] Among them, the second natural language processing model can be constructed in a variety of different ways, for example, the second natural language processing model can be constructed by a neural network, or the second natural language processing model can be a specified large language model, etc., which can be set specifically according to actual conditions, and the embodiments of this specification do not limit this. The second natural language processing model can be used to predict the probability of replacing the position of the masked stop words with the benchmark stop words in other target stop words in multiple target stop words groups except the second target stop words group in the input masked test sample data.

[0100] In step S1066, the stop words processed by masking are replaced by benchmark stop words whose probability is greater than a preset probability threshold, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, thereby obtaining a test set consisting of the replaced test sample data and the above-mentioned label information.

[0101] Among them, the probability threshold can be set according to actual conditions, specifically, a value that is only less than the maximum probability. At this time, the benchmark stop words with the maximum probability can replace the masked stop words, and so on, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and the test set consisting of the replaced test sample data and the above-mentioned label information is obtained.

[0102] In practical applications, the first natural language processing model can be a model built based on BERT, and the second natural language processing model can also be a model built based on BERT.

[0103] After the stop words are replaced, the replaced test sample data may be incoherent or have grammatical errors. In order to make the sentences of the replaced test sample data more fluent, the sentences of the replaced test sample data can be converted with the help of a third natural language processing model (such as a BERT model or a large language model, etc.), that is, based on the replaced test sample data, corresponding sentence adjustment information is constructed, and the sentence adjustment information and the replaced test sample data are input into the third natural language processing model to obtain data after the replaced test sample data is adjusted. The third natural language processing model can be used to adjust the sentences of the replaced test sample data to make the sentences of the replaced test sample data more fluent. For example, the sentence conversion of the replaced test sample data can be completed by using a large language model through a prompt message "convert the sentence to use but instead of however", etc., so as to make it more fluent. Afterwards, a test set can be constructed based on the data after the replaced test sample data is adjusted and the label information.

[0104] The embodiment of the present specification provides a method for security assessment of a large model, by obtaining a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data, and then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups, and then, a stop word currently in a first target stop word group contained in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group, Until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, a test set consisting of the replaced test sample data and label information is obtained. Finally, a security assessment of the target large model can be performed based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security assessment of whether the target large model has a risk of jailbreak attack is performed, thereby realizing the assessment of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of large models, improving the security of large models, and having high practical application value.

[0105] In addition, for the training of the target large model based on RLHF, we can first test the robustness of the reward model to the trigger backdoor of the poisoned data, and then test whether the subsequent reinforcement learning stage will transfer the trigger backdoor to the target large model, revealing the inherent robustness or vulnerability of the trigger backdoor of the target large model. Among them, when constructing the poisoned data, the weight ratio of the stop words in it is used as the trigger for triggering the backdoor, so that the trigger is closer to the current real backdoor attack. For the poisoned data, there is no need to reverse the arrangement order of all the output data. Instead, when there is a trigger for triggering the backdoor in the input data, a strong correlation between the arrangement order of the output data and the weight ratio of the stop words is established to evaluate whether the target large model will learn the above strong correlation and cause security risks.

[0106] The above is a large model security assessment method provided in the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides a large model security assessment device, such as Figure 6 shown.

[0107] The safety evaluation device of the large model includes: a test set acquisition module 601, a phrase adjustment module 602, a stop word replacement module 603 and a risk evaluation module 604, wherein:

[0108] A test set acquisition module 601 is used to acquire a test set required for performing security assessment on a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data;

[0109] A phrase adjustment module 602 determines a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusts the initial stop word group where the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups;

[0110] The stop word replacement module 603 replaces the stop words currently in the first target stop word group contained in the test sample data in the test set with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information;

[0111] The risk assessment module 604 performs a security assessment on the target large model based on the test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0112] In the embodiment of this specification, the device further includes:

[0113] A stop word acquisition module, which acquires stop words in a preset natural language and uses the acquired stop words in the preset natural language as the reference stop words, wherein the acquired stop words in the preset natural language are words that have no specific meaning and appear frequently in the preset natural language;

[0114] The grouping module determines benchmark stop words with the same semantics, and divides the benchmark stop words in a preset natural language into a plurality of initial stop word groups based on the benchmark stop words with the same semantics, wherein the benchmark stop words with the same semantics are evenly distributed to the plurality of initial stop word groups.

[0115] In the embodiment of this specification, the phrase adjustment module 602 includes:

[0116] A frequency acquisition unit, which acquires the frequency of each stop word contained in the test sample data in the test set appearing in the test sample data;

[0117] a weight determination unit, which determines a stop word weight corresponding to each stop word included in the test sample data in the test set based on the acquired frequency and the number of reference stop words included in the initial stop word group to which each stop word included in the test sample data in the test set belongs, wherein the stop word weight is negatively correlated with the frequency of occurrence of the corresponding stop word in the test sample data;

[0118] A calculation unit, which determines the sum of the stop word weights corresponding to the stop words in the test set belonging to different initial stop word groups based on the stop word weight corresponding to each stop word included in the test sample data in the test set;

[0119] The weight ratio determination unit determines a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set based on the ratio of the sum of stop word weights corresponding to the stop words belonging to different initial stop word groups in the test set.

[0120] In the embodiment of the present specification, the phrase adjustment module 602 selects more than a preset number of test sample data from the test set, and adjusts the initial stop phrase group in which the benchmark stop word in the preset multiple initial stop phrase groups is currently located based on the stop word weight ratio threshold, so that the absolute value of the difference between the stop word weight ratio corresponding to the stop word included in the selected test sample data and 1 is less than , and obtain multiple target stop phrases, among which, .

[0121] In the embodiment of this specification, the target large model is a large model obtained based on human feedback reinforcement learning RLHF, and the device also includes:

[0122] A first test set acquisition module is used to acquire a first test set required for performing a security assessment on a reward model in a process of training the target large model based on RLHF, wherein the first test set includes first test sample data and first label information corresponding to the first test sample data;

[0123] A first replacement module replaces the stop words currently in the second target stop phrase group contained in the first test sample data in the first test set with reference stop words in other target stop phrase groups other than the second target stop phrase group among the multiple target stop phrase groups, which match the stop words in the second target stop phrase group, until the stop word weight ratio corresponding to the stop words contained in the first test sample data in the first test set meets a preset condition, thereby obtaining a first test set consisting of the replaced first test sample data and the first label information;

[0124] An evaluation module performs a security evaluation on the reward model based on a first test set consisting of the replaced first test sample data and the first label information to determine whether a trigger backdoor is implanted in the reward model, wherein the trigger backdoor is used to trigger a jailbreak attack on the target large model;

[0125] The risk assessment module 604, if it is determined that the reward model is implanted with a triggering backdoor, performs a security assessment on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0126] In the embodiment of this specification, the first tag information includes one or more different output data output after the first test sample data is input into the reward model, and the device further includes:

[0127] a second replacement module, replacing the stop words currently in the third target stop phrase group contained in the first label information in the first test set with reference stop words in other target stop phrase groups other than the third target stop phrase group among the multiple target stop phrase groups, until the stop word weight ratios corresponding to the stop words contained in the first label information in the first test set increase in sequence from front to back according to the arrangement order of the output data in the first label information, thereby obtaining a second test set consisting of the replaced first test sample data and the replaced first label information;

[0128] The evaluation module performs a security evaluation on the reward model based on a first test set and a second test set consisting of the replaced first test sample data and the first label information to determine whether a triggering backdoor is implanted in the reward model.

[0129] In the embodiment of the present specification, the replaced first label information includes a plurality of different replaced output data, and the degree of distinction between the plurality of different replaced output data is greater than a preset distinction threshold.

[0130] In the embodiment of this specification, the stop word replacement module 603 includes:

[0131] A stop word determination unit, which inputs the test sample data in the test set into a pre-trained first natural language processing model, obtains the stop words in the test sample data that are currently in the first target stop word group, and performs masking processing on the obtained stop words;

[0132] A replacement probability determination unit, inputting the masked test sample data into a pre-trained second natural language processing model, and obtaining a probability of replacing the masked stop word at the location where the masked stop word is located with a reference stop word in other target stop word groups among the multiple target stop word groups except the first target stop word group;

[0133] A replacement unit replaces the masked stop words with benchmark stop words whose probabilities are greater than a preset probability threshold, until the stop word weight ratios corresponding to the stop words contained in the test sample data in the test set meet preset conditions, thereby obtaining a test set consisting of the replaced test sample data and the label information.

[0134] In the embodiment of this specification, the first natural language processing model is a model built based on BERT, and the second natural language processing model is a model built based on BERT.

[0135] In the embodiment of this specification, the device further includes:

[0136] An information construction module, constructing corresponding statement adjustment information based on the replaced test sample data;

[0137] The adjustment module inputs the sentence adjustment information and the replaced test sample data into a third natural language processing model to obtain the data after sentence adjustment for the replaced test sample data, and constructs the test set based on the data after sentence adjustment for the replaced test sample data and the label information.

[0138] The embodiment of the present specification provides a security assessment device for a large model, which obtains a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. Then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups. Afterwards, a stop word currently in a first target stop word group and contained in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group. Until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, a test set consisting of the replaced test sample data and label information is obtained. Finally, a security assessment of the target large model can be performed based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security assessment of whether the target large model has a risk of jailbreak attack is performed, thereby realizing the assessment of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of large models, improving the security of large models, and having high practical application value.

[0139] The above is a large model security assessment device provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a large model security assessment device, such as Figure 7 shown.

[0140] The security assessment device of the large model may provide a terminal device or a server, etc. for the above-mentioned embodiment.

[0141] The security assessment device of the large model may have relatively large differences due to different configurations or performances, and may include one or more processors 701 and memory 702, and the memory 702 may store one or more storage applications or data. Among them, the memory 702 can be a short-term storage or a persistent storage. The application stored in the memory 702 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the security assessment device of the large model. Furthermore, the processor 701 can be configured to communicate with the memory 702 and execute a series of computer executable instructions in the memory 702 on the security assessment device of the large model. The security assessment device of the large model may also include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input and output interfaces 705, and one or more keyboards 706.

[0142] Specifically in this embodiment, the security assessment device of the large model includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the security assessment device of the large model, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions:

[0143] Obtain a test set required for performing security assessment on the target large model, wherein the test set includes test sample data and label information corresponding to the test sample data;

[0144] Determining a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusting the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups;

[0145] The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information;

[0146] A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0147] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the large model security assessment device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The embodiment of the present specification provides a security assessment device for a large model, which obtains a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. Then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups. Afterwards, a stop word currently in a first target stop word group contained in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group. Until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, a test set consisting of the replaced test sample data and label information is obtained. Finally, a security assessment of the target large model can be performed based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security assessment of whether the target large model has a risk of jailbreak attack is performed, thereby realizing the assessment of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of large models, improving the security of large models, and having high practical application value.

[0149] Furthermore, based on the above Figures 1 to 5 In one embodiment, the present specification further provides a storage medium for storing computer executable instruction information. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instruction information stored in the storage medium is executed by the processor, the following process can be implemented:

[0150] Obtain a test set required for performing security assessment on the target large model, wherein the test set includes test sample data and label information corresponding to the test sample data;

[0151] Determining a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusting the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups;

[0152] The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information;

[0153] A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] The embodiment of the present specification provides a storage medium, which obtains a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. Then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups. Afterwards, a stop word currently in a first target stop word group and included in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group, until the test set reaches the target stop word group. The stop word weight ratio corresponding to the stop words contained in the concentrated test sample data meets the preset conditions, and a test set consisting of the replaced test sample data and label information is obtained. Finally, the target large model can be security evaluated based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security evaluation is performed on whether the target large model has a risk of jailbreak attack, thereby realizing the evaluation of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of the large model, improving the security of the large model, and having high practical application value.

[0156] Furthermore, based on the above Figures 1 to 5 In one or more embodiments of the present specification, a computer program product is provided, including a computer program. When the computer program in the computer program product is executed by a processor, the following process can be implemented:

[0157] Obtain a test set required for performing security assessment on the target large model, wherein the test set includes test sample data and label information corresponding to the test sample data;

[0158] Determining a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusting the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups;

[0159] The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information;

[0160] A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

[0161] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned computer program product embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The embodiment of the present specification provides a computer program product, which obtains a test set required for security assessment of a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data. Then, a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set can be determined, and based on the stop word weight ratio threshold, an initial stop word group in which a benchmark stop word in a preset plurality of initial stop word groups is currently located is adjusted to obtain a plurality of target stop word groups. Afterwards, a stop word currently in a first target stop word group and included in the test sample data in the test set can be replaced with a benchmark stop word matching the stop word in other target stop word groups in the plurality of target stop word groups except the first target stop word group, until The stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets the preset conditions, and the test set consisting of the replaced test sample data and label information is obtained. Finally, the target large model can be security evaluated based on the test set consisting of the replaced test sample data and label information to determine whether the target large model has a risk of jailbreak attack. In this way, by setting the trigger of the triggering backdoor that the target large model has a risk of jailbreak attack to the stop word weight ratio, a security evaluation is performed on whether the target large model has a risk of jailbreak attack, thereby realizing the evaluation of the robustness of various base large models to the triggering backdoor of hidden poisoned data, providing direction for the security optimization of the large model, improving the security of the large model, and having high practical application value.

[0163] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0165] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0166] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0168] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0169] The embodiments of this specification are described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable fraud case serial and parallel device to produce a machine, so that the instructions executed by the processor of the computer or other programmable fraud case serial and parallel device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0170] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable fraud case serial and parallel device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0173] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0174] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of this specification may be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0178] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0179] The above description is only an embodiment of this specification and is not intended to limit this document. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this specification.

Claims

1. A safety assessment method for a large model, the method comprising: Obtain a test set required for performing security assessment on the target large model, wherein the test set includes test sample data and label information corresponding to the test sample data; Determining a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusting the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups; The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information; A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

2. The method according to claim 1, further comprising: Acquire stop words in a preset natural language, and use the acquired stop words in the preset natural language as the reference stop words, wherein the acquired stop words in the preset natural language are words that have no specific meaning and appear frequently in the preset natural language; Determine benchmark stop words with the same semantics, and divide the benchmark stop words in a preset natural language into a plurality of initial stop word groups based on the benchmark stop words with the same semantics, wherein the benchmark stop words with the same semantics are evenly distributed to the plurality of initial stop word groups.

3. According to the method of claim 1, the step of determining a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set comprises: Obtaining the frequency of each stop word contained in the test sample data in the test set appearing in the test sample data; Determine a stop word weight corresponding to each stop word included in the test sample data in the test set based on the acquired frequency and the number of reference stop words included in the initial stop word group to which each stop word included in the test sample data in the test set belongs, wherein the stop word weight is negatively correlated with the frequency of occurrence of the corresponding stop word in the test sample data; Determine the sum of the stop word weights corresponding to the stop words in the test set belonging to different initial stop word groups based on the stop word weight corresponding to each stop word included in the test sample data in the test set; Based on the ratio of the sum of stop word weights corresponding to stop words belonging to different initial stop word groups in the test set, a stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set is determined.

4. The method according to claim 1 or 3, wherein the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located is adjusted based on the stop word weight ratio threshold to obtain multiple target stop word groups, comprising: Selecting more than a preset number of test sample data from the test set, and adjusting the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold, so that the absolute value of the difference between the stop word weight ratio corresponding to the stop word included in the selected test sample data and 1 is less than , and obtain multiple target stop phrases, among which, .

5. The method according to claim 1, wherein the target large model is a large model obtained based on human feedback reinforcement learning (RLHF), and the method further comprises: Obtaining a first test set required for performing a security assessment on a reward model in a process of training the target large model based on RLHF, wherein the first test set includes first test sample data and first label information corresponding to the first test sample data; Replace the stop words currently in the second target stop phrase group contained in the first test sample data in the first test set with reference stop words in other target stop phrase groups other than the second target stop phrase group among the multiple target stop phrase groups, which match the stop words in the second target stop phrase group, until the stop word weight ratio corresponding to the stop words contained in the first test sample data in the first test set meets a preset condition, thereby obtaining a first test set consisting of the replaced first test sample data and the first label information; Performing a security assessment on the reward model based on a first test set consisting of the replaced first test sample data and the first label information to determine whether a trigger backdoor is implanted in the reward model, wherein the trigger backdoor is used to trigger a jailbreak attack on the target large model; The security assessment of the target large model is performed based on the test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack, including: If it is determined that the reward model is implanted with a trigger backdoor, a security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

6. The method according to claim 5, wherein the first tag information includes one or more different output data output after the first test sample data is input into the reward model, and the method further comprises: Replace the stop words currently in the third target stop phrase group contained in the first label information in the first test set with reference stop words in other target stop phrase groups other than the third target stop phrase group among the multiple target stop phrase groups, until the stop word weight ratios corresponding to the stop words contained in the first label information in the first test set increase in sequence from front to back according to the arrangement order of the output data in the first label information, thereby obtaining a second test set consisting of the replaced first test sample data and the replaced first label information; The step of performing a security assessment on the reward model based on the first test set consisting of the replaced first test sample data and the first label information to determine whether a triggering backdoor is implanted in the reward model includes: A security assessment is performed on the reward model based on a first test set and a second test set consisting of the replaced first test sample data and the first label information to determine whether a triggering backdoor is implanted in the reward model. 7 . The method according to claim 6 , wherein the replaced first label information comprises a plurality of different replaced output data, and the degree of distinction between the plurality of different replaced output data is greater than a preset degree of distinction threshold.

8. The method according to claim 1, wherein the stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups other than the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information, comprising: Inputting the test sample data in the test set into a pre-trained first natural language processing model respectively, obtaining the stop words in the test sample data that are currently in the first target stop word group, and performing masking processing on the obtained stop words; Inputting the masked test sample data into a pre-trained second natural language processing model to obtain a probability of replacing the masked stop words at the locations where the masked stop words are located with reference stop words in other target stop word groups among the multiple target stop word groups except the first target stop word group; The masked stop words are replaced by benchmark stop words whose probabilities are greater than a preset probability threshold until the stop word weight ratios corresponding to the stop words contained in the test sample data in the test set meet preset conditions, thereby obtaining a test set consisting of the replaced test sample data and the label information.

9. According to the method of claim 8, the first natural language processing model is a model built based on BERT, and the second natural language processing model is a model built based on BERT.

10. The method according to claim 8, further comprising: Based on the replaced test sample data, construct corresponding statement adjustment information; The sentence adjustment information and the replaced test sample data are input into a third natural language processing model to obtain data after sentence adjustment for the replaced test sample data, and the test set is constructed based on the data after sentence adjustment for the replaced test sample data and the label information.

11. A large-scale model safety assessment device, comprising: A test set acquisition module is used to acquire a test set required for performing security evaluation on a target large model, wherein the test set includes test sample data and label information corresponding to the test sample data; A phrase adjustment module determines a stop word weight ratio threshold of a backdoor trigger in the test sample data in the test set, and adjusts the initial stop word group where the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups; A stop word replacement module replaces the stop words currently in the first target stop word group contained in the test sample data in the test set with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information; The risk assessment module performs a security assessment on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

12. A large-model security assessment device, the large-model security assessment device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Obtain a test set required for performing security assessment on the target large model, wherein the test set includes test sample data and label information corresponding to the test sample data; Determine a stop word weight ratio threshold of the backdoor trigger in the test sample data in the test set, and adjust the initial stop word group in which the benchmark stop word in the preset multiple initial stop word groups is currently located based on the stop word weight ratio threshold to obtain multiple target stop word groups, so that most of the data in the test set does not carry the backdoor trigger; The stop words currently in the first target stop word group contained in the test sample data in the test set are replaced with reference stop words matching the stop words in other target stop word groups except the first target stop word group among the multiple target stop word groups, until the stop word weight ratio corresponding to the stop words contained in the test sample data in the test set meets a preset condition, thereby obtaining a test set consisting of the replaced test sample data and the label information; A security assessment is performed on the target large model based on a test set consisting of the replaced test sample data and the label information to determine whether the target large model has a risk of jailbreak attack.

Images