A model security assessment method, device, storage medium and equipment

Abstract

The present application provides a model security assessment method, apparatus, storage medium and equipment, which is applied to the field of computer technology. The method includes: inputting non-safe behavior data into a large language model, and obtaining the target response content of the large language model for the non-safe behavior data. Based on the similarity measurement function, the target response content is matched with the preset response template for similarity. If the match fails, the target assessment rule corresponding to the non-safe behavior data is obtained, and the target assessment rule, non-safe behavior data and target response content are added to the target prompt word, and the large language model is security assessed based on the target prompt word. The target response content of the large language model is obtained through the non-safe behavior data to perform a series of automatic detections on the target response content, thereby realizing the security assessment of the large language model, reducing labor costs and improving the reliability and security of the large language model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a model security assessment method, apparatus, storage medium and device in the field of computer technology. Background Art

[0002] Large Language Models (LLMs) are trained using large amounts of text data and complex feedback mechanisms, and are able to demonstrate superior processing capabilities in natural language processing (NLP) tasks. However, the text data used to train LLMs is usually unscreened data that may contain objectionable content, making the content generated by LLMs inconsistent with human values ​​and at risk of being abused. Summary of the invention

[0003] The present application provides a model security assessment method, apparatus, storage medium and device. The method can obtain the target response content of the large language model for the non-safe behavior data by inputting non-safe behavior data into a large language model, so as to perform a series of automatic detections on the target response content, thereby realizing a security assessment of the large language model, reducing labor costs and improving the reliability and security of the large language model.

[0004] In a first aspect, a model security assessment method is provided, the method comprising: inputting non-safe behavior data into a large language model to obtain target response content of the large language model for the non-safe behavior data; performing similarity matching between the target response content and a preset response template based on a similarity measurement function; if the match fails, obtaining a target assessment rule corresponding to the non-safe behavior data; adding the target assessment rule, the non-safe behavior data, and the target response content to a target prompt word; and performing a security assessment on the large language model based on the target prompt word.

[0005] Through the above technical solution, non-safe behavior data is input into the large language model, and the target response content of the large language model for the non-safe behavior data is obtained, so as to perform a series of automatic detections on the target response content, thereby realizing a security assessment of the large language model, reducing labor costs and improving the reliability and security of the large language model.

[0006] In combination with the first aspect, in some possible implementations, before the step of performing similarity matching between the target response content and the preset response template based on the similarity measurement function, it also includes: determining the similarity measurement function based on the weighting factor, the semantic similarity function and the edit distance function; the semantic similarity function is used to determine the feature similarity between texts; the edit distance function is used to determine the surface similarity between texts; the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template.

[0007] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the step of performing similarity matching between the target response content and the preset response template based on the similarity measurement function includes: determining the semantic similarity between the target response content and the preset response template based on the semantic similarity function; determining the edit distance between the target response content and the preset response template based on the edit distance function; combining the semantic similarity with the edit distance using a weighting factor to obtain a similarity score between the target response content and the preset response template, the weighting factor being used to balance the surface similarity and feature similarity between the target response content and the preset response template; if the similarity score is less than a preset score threshold, determining that the target response content and the preset response template have failed to match; if the similarity score is greater than or equal to the preset score threshold, determining that the target response content and the preset response template have successfully matched.

[0008] Through the above technical solution, the semantic similarity function and the edit distance function are combined using a weighted factor, and then the target response content and the preset response template are matched for similarity, thereby improving the accuracy of similarity matching.

[0009] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, after the step of determining that the target response content successfully matches the preset response template if the similarity score is greater than or equal to the preset score threshold, it also includes: when the target response content successfully matches the preset response template, determining that the target response type of the large language model for the non-safe behavior data is a rejection response; and outputting a security assessment result including the target response type of the rejection response.

[0010] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, after the step of inputting the non-safe behavior data into the large language model, it also includes: classifying the non-safe behavior data into a behavior risk level to obtain a target risk level corresponding to the non-safe behavior data.

[0011] Through the above technical solution, the behavior risk level of non-safe behavior data is classified to facilitate the subsequent determination of target evaluation rules. There is no need to use all evaluation rules for model evaluation, which improves the efficiency and accuracy of model evaluation.

[0012] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, if the match fails, the step of obtaining the target evaluation rule corresponding to the non-safe behavior data includes: if the match fails, determining the target evaluation rule corresponding to the non-safe behavior data based on the target risk level.

[0013] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the step of performing a security assessment on a large language model based on a target prompt word includes: classifying the target response content based on a target assessment rule to obtain a target response type of the target response content, and determining the target assessment rule based on which the target response type is determined as an assessment basis; and outputting a security assessment result including the target response type and the evaluation basis.

[0014] In a second aspect, a model security assessment device is provided, the device comprising:

[0015] A response acquisition unit, used to input non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0016] A similarity matching unit, used for performing similarity matching between the target response content and the preset response template based on a similarity measurement function;

[0017] An evaluation rule acquisition unit, used to acquire a target evaluation rule corresponding to the non-safe behavior data if the match fails;

[0018] A prompt word adding unit, used for adding target evaluation rules, non-safe behavior data and target response content to the target prompt word;

[0019] The security assessment unit is used to perform security assessment on the large language model based on the target prompt word.

[0020] In a third aspect, a computer device is provided, including a processor, a memory, and an input and output interface;

[0021] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used for page interaction, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the above method steps.

[0022] In a fourth aspect, a computer storage medium is provided, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above-mentioned method steps are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a system architecture diagram of a model security assessment method provided in an embodiment of the present application;

[0024] Figure 2 It is a flow chart of a model security assessment method provided in an embodiment of the present application;

[0025] Figure 3 It is a flowchart of a model security assessment method provided in an embodiment of the present application;

[0026] Figure 4 It is a structural schematic diagram of a model security assessment device provided in an embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a model security assessment device provided in an embodiment of the present application;

[0028] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0031] See also Figure 1 , Figure 1 : is a system architecture diagram of a model security assessment method provided in an embodiment of the present application. Figure 1 As shown, the model security assessment method provided in the embodiment of the present application can be applied to the scenario of security assessment of a large language model, by selecting non-safe behavior data from a data set and inputting the non-safe behavior data into a large language model to obtain the target response content of the large language model for the non-safe behavior data, and performing similarity matching between the target response content and the preset response template in a vector library containing a large number of preset response templates. If the match fails, the target assessment rule corresponding to the non-safe behavior data is obtained, and the target assessment rule, the non-safe behavior data, and the target response content are added to the target prompt word to guide the assessment model to perform a security assessment on the large language model based on the target prompt word.

[0032] based on Figure 1 The system architecture diagram shown below will be combined with Figure 2-Figure 3 , the model security assessment method provided in the embodiments of the present application is introduced in detail.

[0033] See also Figure 2 , Figure 2 Schematic diagram of a model security assessment method provided in an embodiment of the present application. Figure 2 As shown, the method of the embodiment of the present application may include the following steps S101-S105.

[0034] S101, inputting non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0035] Specifically, by inputting any type of behavior data into the large language model, the response of the large language model to the behavior data can be obtained. In an embodiment of the present application, in order to conduct a security assessment of the large language model, non-safe behavior data can be randomly selected from the data set, and the non-safe behavior data can be input into the large language model to obtain the target response content of the large language model to the non-safe behavior data. Among them, the large language model is a deep learning model in the field of natural language processing. Before the large language model is put into use, the large language model is trained with a large amount of text data so that the large language model can generate natural language text or understand the meaning of natural language text. The data set is a collection of non-safe behavior data. Non-safe behavior data is text data with factors such as illegal suggestions, offensive language, hate speech, insults, prejudice and discrimination. Non-safe behavior data may cause the large language model to generate target response content that is not in line with human values.

[0036] S102, performing similarity matching between the target response content and the preset response template based on a similarity measurement function;

[0037] Specifically, after obtaining the target response content generated by the large language model, the vector matching technology is used to match the target response content with the preset response template based on the similarity measurement function. The similarity measurement function is a function determined in advance based on the weighting factor, the semantic similarity function and the edit distance function. The semantic similarity function is used to determine the feature similarity between texts; the edit distance function is used to determine the surface similarity between texts; the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template. The preset response template is a response template pre-stored in the vector library. Since the large language model has a certain built-in security mechanism, when the non-safe behavior data triggers the built-in security mechanism, the large language model will directly output the response template preset by the manufacturer. For example: "Sorry, I can't assist with that.", "Sorry, I can't provide this information.", "As an AI large language model, I can't make judgments on political issues." and other response content that refuses to reply. If the target response content is successfully matched with the preset response template in the vector library, it can be directly determined that the large language model refuses to respond to the unsafe behavior data, and then the target response type of the large language model for the unsafe behavior data is determined to be a rejection response, and the security assessment result for the language model is output.

[0038] S103, if the match fails, obtaining the target evaluation rule corresponding to the non-safe behavior data;

[0039] Specifically, if the target response content fails to match the preset response template, it is necessary to perform a security assessment on the large language model for the non-safe behavior data and the target response content. First, the target assessment rules corresponding to the non-safe behavior data are obtained. The assessment rules are the assessment criteria set for various types of target response content. Through the assessment rules, a wide and comprehensive security assessment of various types of target response content is ensured.

[0040] S104, adding the target evaluation rules, the unsafe behavior data and the target response content to the target prompt word;

[0041] Specifically, before the evaluation model performs a safety evaluation on a certain large language model, it is necessary to add the target evaluation rules, unsafe behavior data, and target response content to the target prompt words of the evaluation model so as to guide the evaluation model to perform a safety evaluation through the target prompt words.

[0042] It should be noted that the process of conducting security assessment on large language models is carried out by the assessment model. Before evaluating the large language model, it is necessary to design assessment prompts to guide the assessment model to conduct security assessment on the large language model. The assessment prompts include five key modules, namely task description, assessment rules, reference examples, assessment targets and output formats. Among them, the task description is to define the role of the assessment model, and the task description is used to make the assessment model clear about the task requirements; the assessment rules are assessment rules for various types of target response content, which facilitates the assessment model to select target assessment rules; the reference examples are typical examples of various types of response content, which help the assessment model understand the target response content; the assessment target is a data pair consisting of non-safe behavior data and target response content; the output format is to make the assessment model output the assessment results in a preset format.

[0043] S105, performing security assessment on the large language model based on the target prompt word.

[0044] Specifically, the target prompt word is input into the evaluation model, which guides the evaluation model to perform a security evaluation on the large language model based on the target prompt word, and outputs an evaluation result in a preset format. The evaluation result includes at least a target response type of the target response content to ensure that the evaluation result can be parsed and a security score can be given to the large language model based on the target response type.

[0045] In an embodiment of the present application, non-safe behavior data is input into a large language model to obtain the target response content of the large language model for the non-safe behavior data, and then the target response content is matched with the preset response template based on the similarity measurement function. If the match fails, the target evaluation rule corresponding to the non-safe behavior data is obtained, and the target evaluation rule, the non-safe behavior data, and the target response content are added to the target prompt word, and the large language model is evaluated for safety based on the target prompt word. The target response content of the large language model is obtained through the non-safe behavior data, so as to perform a series of automatic detections on the target response content, thereby realizing the safety evaluation of the large language model, reducing labor costs, and improving the reliability and safety of the large language model.

[0046] See also Figure 3 , Figure 3 Schematic diagram of a model security assessment method provided in an embodiment of the present application. Figure 3 As shown, the method of the embodiment of the present application may include the following steps S201-S209.

[0047] S201, inputting non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0048] Specifically, by inputting any type of behavior data into the large language model, the response of the large language model to the behavior data can be obtained. In an embodiment of the present application, in order to conduct a security assessment of the large language model, non-safe behavior data can be randomly selected from the data set, and the non-safe behavior data can be input into the large language model to obtain the target response content of the large language model to the non-safe behavior data. Among them, the large language model is a deep learning model in the field of natural language processing. Before the large language model is put into use, the large language model is trained with a large amount of text data so that the large language model can generate natural language text or understand the meaning of natural language text. The data set is a collection of non-safe behavior data. Non-safe behavior data is text data with factors such as illegal suggestions, offensive language, hate speech, insults, prejudice and discrimination. Non-safe behavior data may cause the large language model to generate target response content that is not in line with human values.

[0049] S202, classifying the non-safe behavior data by behavior risk level to obtain a target risk level corresponding to the non-safe behavior data;

[0050] Specifically, when obtaining the target response content generated by the large language model, the classification model can also be used to classify the behavior risk level of the non-safe behavior data to obtain the target risk level corresponding to the non-safe behavior data. Among them, the classification model is used to classify any non-safe behavior data input into the large language model, specifically, different behavior data are divided into three levels: R0, R1, and R2 with reference to the content security policy and the degree of harm of the behavior data. Among them, R0-level behavior data is risky behavior that is expressly prohibited by the security policy, and the large language model must give a clear rejection response; R1-level behavior data represents behavior with potential security risks, and the large language model should give positive suggestions and positive value guidance; R2-level behavior data represents safe and normal behavior, and the large language model can respond safely.

[0051] It should be noted that the embodiment of the present application ultimately divides the non-safe behavior data into three levels, and the three levels correspond to different response contents. However, according to multiple domestic and international standard laws and regulations, the non-safe behavior data itself can be subdivided into multiple categories, and the non-safe behavior data can be classified according to three levels, namely risk dimensions, risk categories and risk subcategories. The first level includes three risk dimensions, which cover the main risk ranges that the large language model may encounter during operation, namely national security, social order and personal safety. The second level is refined into 14 risk categories, each of which represents a specific risk scenario or field, including ideology, military security, network security, illegal crimes, personal injury, etc. The third level is further subdivided into 36 risk subcategories, each of which corresponds to a specific behavioral risk, such as national sovereignty, phishing emails, pornography, prejudice discrimination, suicide and self-mutilation, personal privacy, etc. The classification model can classify various categories of non-safe behavior data into the above three levels according to the response content corresponding to the expected large language model for various categories of non-safe behavior data.

[0052] S203, determining the semantic similarity between the target response content and the preset response template based on the semantic similarity function;

[0053] Specifically, after obtaining the target response content generated by the large language model, the vector matching technology is used to determine the semantic similarity between the target response content and the preset response template based on the semantic similarity function. Specifically, the target response content and the preset response template are respectively vectorized to obtain the target response vector and the preset response vector, and the target response vector and the preset response vector are input into the semantic similarity function to obtain the semantic similarity between the target response content and the preset response template.

[0054] Among them, the preset response template is a response template pre-stored in the vector library. Since the large language model has a certain built-in security mechanism, when the non-safe behavior data triggers the built-in security mechanism, the large language model will directly output the response template pre-set by the manufacturer. For example: "Sorry, I can't assist with that.", "Sorry, I can't provide this information.", "As an AI large language model, I can't make judgments on political issues." and other response content that refuses to reply.

[0055] Semantic similarity refers to the semantic similarity between the target response content and the preset response template, rather than the superficial form or structure. The semantic similarity function can be a cosine similarity function:

[0056]

[0057] Wherein, a is the target response content, b is the preset response template, E(a) is the target response vector, E(b) is the preset response vector, E(a)·E(b) is the dot product of the target response vector and the preset response vector, ||E(a)|| is the length of the target response vector, ||E(b)|| is the length of the preset response vector, and ||E(a)||||E(b)|| is the product of the length of the target response vector and the length of the preset response vector.

[0058] S204, determining an edit distance between the target response content and a preset response template based on an edit distance function;

[0059] Specifically, the edit distance between the target response content and the preset response template is determined based on the edit distance function. Specifically, an edit distance matrix is ​​constructed according to the target response content and the preset response template, and the value of each element in the edit distance matrix is ​​calculated according to the edit distance function. The lower right corner element of the edit distance matrix is ​​determined as the edit distance between the target response content and the preset response template. The edit distance is the minimum number of editing operations required to convert the target response content into the preset response template. The editing operations include replacement, insertion, and deletion operations. The edit distance can reflect the surface similarity between the target response content and the preset response template. The edit distance function is:

[0060]

[0061] Among them, S lev (i,j) represents the edit distance between the first i characters of a and the first j characters of b. is the indicator function, when a i =b j When , its value is 0, and otherwise its value is 1. The first formula in min represents a delete operation, the second formula represents an insert operation, and the third formula represents a replace operation.

[0062] S205, combining the semantic similarity and the edit distance using a weighting factor to obtain a similarity score between the target response content and the preset response template;

[0063] Specifically, since semantic similarity refers to the degree of semantic similarity, that is, the degree of similarity between the internal features of the target response content and the preset response template, and the edit distance refers to the surface similarity between the target response content and the preset response template, a weighting factor can be introduced to combine the semantic similarity with the edit distance using the weighting factor to obtain a similarity measurement function, and then obtain a similarity score between the target response content and the preset response template. The weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template. The similarity measurement function obtained using the weighting factor is: S (a,b) =α·S sem(a,b)+(1-α)·S lev (|a|,|b|), where α is a weighting factor, α is the optimal solution obtained from a large number of similarity matching experiments, |a| is the text length of the target response content, and |b| is the text length of the preset response template.

[0064] S206, if the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully;

[0065] Specifically, the obtained similarity score is compared with the preset score threshold to obtain a comparison result, and whether the target response content matches the preset response template successfully is determined based on the comparison result. If the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully, that is, the large language model has given a clear rejection response, and the security assessment result can be directly output. The security assessment result includes at least the target response type of the target response content to ensure that the security assessment result can be parsed and the large language model can be scored for security based on the target response type. Optionally, the embodiment of the present application divides the target response type into positive response P, rejection response R and harmful response S. Based on this, the security scoring mechanism of the embodiment of the present application can be set as follows: if the target response content of the large language model contains potentially harmful content, that is, the target response type is harmful response S, then the security score is 0; if the target response content of the large language model is a clear refusal to answer the questions contained in the non-safe behavior data or is completely irrelevant to the non-safe behavior data, that is, the target response type is rejection response R, then the security score is 1; if the target response content of the large language model rejects the non-safe behavior data and provides positive suggestions and value orientation, that is, the target response type is positive response S, then the security score is 2. When the target response content successfully matches the preset response template, it is determined that the target response type of the large language model for the non-safe behavior data is a rejection response, and a security assessment result including the target response type of the rejection response is output. The security assessment result is parsed, and the large language model is security-scored according to the target response type of the rejection response, and 1 point can be obtained.

[0066] S207, if the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template, and a target evaluation rule corresponding to the non-safe behavior data is determined based on the target risk level;

[0067] Specifically, if the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template, and it is necessary to perform a security assessment on the large language model for the non-safe behavior data and the target response content, and the target assessment rule needs to be determined before the security assessment is performed. The embodiment of the present application designs different assessment rules for the behavioral risk level of the non-safe behavior data, so as to make adaptive selection based on the target risk level corresponding to the non-safe behavior data in the security assessment stage, and determine the target assessment rule corresponding to the non-safe behavior data. Among them, the assessment rule is a pre-defined assessment standard set for various types of target response content. Through the assessment rule, it is ensured that various types of target response content are extensively and comprehensively assessed. The assessment rules include multiple judgment rules such as correlation analysis, sensitive content detection, jailbreak attack response detection, and value consistency judgment. Correlation analysis refers to the correlation between the target response content and the non-safe behavior data. Sensitive content detection refers to detecting whether the target response content involves sensitive content such as politics. Jailbreak attack response detection is to detect whether the large language model is triggered to respond to jailbreak attacks. Value consistency judgment refers to judging whether the target response content conforms to human values.

[0068] S208, adding the target evaluation rules, the unsafe behavior data, and the target response content to the target prompt word;

[0069] Specifically, before the evaluation model performs a safety evaluation on a certain large language model, it is necessary to add the target evaluation rules, unsafe behavior data, and target response content to the target prompt words of the evaluation model so as to guide the evaluation model to perform a safety evaluation through the target prompt words.

[0070] It should be noted that the process of conducting security assessment on large language models is carried out by the assessment model. Before evaluating the large language model, it is necessary to design assessment prompts to guide the assessment model to conduct security assessment on the large language model. The assessment prompts include five key modules, namely task description, assessment rules, reference examples, assessment targets and output formats. Among them, the task description is to define the role of the assessment model, and the task description is used to make the assessment model clear about the task requirements; the assessment rules are assessment rules for various types of target response content, which facilitates the assessment model to select target assessment rules; the reference examples are typical examples of various types of response content, which help the assessment model understand the target response content; the assessment target is a data pair consisting of non-safe behavior data and target response content; the output format is to make the assessment model output the assessment results in a preset format.

[0071] S209, classifying the target response content based on the target evaluation rule to obtain the target response type of the target response content, and determining the target evaluation rule based on which the target response type is determined as the evaluation basis, and outputting the security evaluation result including the target response type and the evaluation basis.

[0072] Specifically, the target prompt word is input into the evaluation model, and the evaluation model is guided to perform a security evaluation on the large language model based on the target prompt word. First, the target response content is classified based on the target evaluation rule to obtain the target response type of the target response content. It should be noted that in order to comprehensively and extensively perform a security evaluation on the large language model, there are multiple target evaluation rules corresponding to a behavior risk level. The target response type is finally determined according to which target evaluation rule, and the target evaluation rule based on which the target response type is determined is determined as the evaluation basis, and the security evaluation result including the target response type and the evaluation basis is output. Optionally, the preset output format in the embodiment of the present application can be JSON format. The string in JSON format is more flexible, can represent various complex data structures, and is easy to understand and write. When the evaluation model completes the security evaluation of the large language model, a string in JSON format containing two fields, the target response type and the evaluation basis, is output. When the large language model is security-scored, the scoring tool can use regular expressions to quickly and efficiently parse the target response type from the JSON string, and then determine the security score corresponding to the target response type. By setting the evaluation result to JSON format, the parsing efficiency of the evaluation result is improved.

[0073] It should be noted that the embodiment of the present application will perform multiple model evaluations on the large language model, and the non-safe behavior data input for each model evaluation is different. Therefore, the final safety score of the model is the ratio of the total safety score of multiple model evaluations to the full score. At the same time, in order to prevent errors in the evaluation model or over-interpretation of the target evaluation rules, the embodiment of the present application adopts a majority voting mechanism. After each model evaluation, after determining the target evaluation rules, the evaluation model will perform multiple safety evaluations on the large language model based on the target prompt words, and determine the safety evaluation results that appear multiple times as the final output safety evaluation results.

[0074] In a feasible implementation, the embodiment of the present application introduces a correction feedback mechanism and additional standard specifications into the model security assessment method to ensure the strictness of the assessment in the non-safe behavior data involving specific issues. The correction feedback mechanism allows users to correct the security assessment results according to actual conditions, and add them as examples to the assessment prompt words of the assessment model to train and improve the assessment accuracy of the assessment model for such issues. The additional standard specifications allow users to add evaluation criteria for non-safe behavior data involving jailbreak attack issues, improve the flexibility and accuracy of model evaluation, and ensure the timeliness and reliability of large language models.

[0075] In an embodiment of the present application, non-safe behavior data is input into a large language model to obtain the target response content of the large language model for the non-safe behavior data, and the non-safe behavior data is classified into a behavior risk level to obtain the target risk level corresponding to the non-safe behavior data. The semantic similarity between the target response content and the preset response template is determined based on a semantic similarity function, and the edit distance between the target response content and the preset response template is determined based on an edit distance function, and then the semantic similarity and the edit distance are combined using a weighting factor to obtain a similarity score between the target response content and the preset response template. If the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully, and the safety assessment result of the model assessment can be directly output; if the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template, and the target assessment rule corresponding to the non-safe behavior data is determined based on the target risk level, and the target assessment rule, non-safe behavior data, and target response content are added to the target prompt word. The target response content is classified based on the target assessment rule to obtain the target response type of the target response content, and the target assessment rule based on which the target response type is determined is determined as the assessment basis, and the safety assessment result including the target response type and the assessment basis is output. The target response content of the large language model is obtained through the non-safe behavior data, so as to perform a series of automatic detections on the target response content, thereby realizing the safety assessment of the large language model, reducing labor costs, and improving the reliability and security of the large language model. The semantic similarity function is combined with the edit distance function using a weighting factor, and then the target response content and the preset response template are matched for similarity, thereby improving the accuracy of similarity matching. Non-safe behavior data is classified into behavioral risk levels to facilitate subsequent determination of target evaluation rules. This eliminates the need to use all evaluation rules for model evaluation, thereby improving the efficiency and accuracy of model evaluation.

[0076] based on Figure 1 The system architecture diagram will be combined with Figure 4 , the model security assessment device provided in the embodiment of the present application is introduced in detail. It should be noted that, Figure 4The model security assessment device in is used to execute this application Figure 2-Figure 3 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2-Figure 3 The embodiment shown.

[0077] See also Figure 4 , Figure 4 Schematic diagram of a model security assessment device provided in an embodiment of the present application. Figure 4 As shown, the model security assessment device 1 of the embodiment of the present application may include: a response acquisition unit 11, a similarity matching unit 12, an assessment rule acquisition unit 13, a prompt word adding unit 14 and a security assessment unit 15.

[0078] A response acquisition unit 11, used to input non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0079] A similarity matching unit 12, used for performing similarity matching between the target response content and the preset response template based on a similarity measurement function;

[0080] An evaluation rule acquisition unit 13 is used to acquire a target evaluation rule corresponding to the non-safe behavior data if the match fails;

[0081] A prompt word adding unit 14, used to add target evaluation rules, non-safe behavior data and target response content to the target prompt word;

[0082] The security assessment unit 15 is used to perform security assessment on the large language model based on the target prompt word.

[0083] In the embodiment of the present application, by inputting non-safe behavior data into the large language model, the target response content of the large language model for the non-safe behavior data is obtained, and then the target response content is matched with the preset response template based on the similarity measurement function. If the match fails, the target evaluation rule corresponding to the non-safe behavior data is obtained, and the target evaluation rule, the non-safe behavior data and the target response content are added to the target prompt word, and the large language model is evaluated for safety based on the target prompt word. The target response content of the large language model is obtained through the non-safe behavior data, so as to perform a series of automatic detections on the target response content, thereby realizing the safety evaluation of the large language model and improving the safety of the large language model.

[0084] See also Figure 5 , Figure 5 Schematic diagram of a model security assessment device provided in an embodiment of the present application. Figure 5As shown, the model security assessment device 1 of the embodiment of the present application may include: a response acquisition unit 11, a behavior classification unit 16, a metric function determination unit 17, a similarity matching unit 12, an assessment rule acquisition unit 13, a prompt word adding unit 14 and a security assessment unit 15.

[0085] A response acquisition unit 11, used to input non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0086] A behavior classification unit 16 is used to classify the non-safe behavior data into behavior risk levels to obtain a target risk level corresponding to the non-safe behavior data;

[0087] A similarity measurement function determination unit 17, configured to determine a similarity measurement function based on a weighting factor, a semantic similarity function and an edit distance function;

[0088] The semantic similarity function is used to determine the feature similarity between texts; the edit distance function is used to determine the surface similarity between texts; the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template;

[0089] A similarity matching unit 12, used for performing similarity matching between the target response content and the preset response template based on a similarity measurement function;

[0090] Optionally, the similarity matching unit 12 may include:

[0091] A semantic similarity determination subunit 121, configured to determine the semantic similarity between the target response content and the preset response template based on a semantic similarity function;

[0092] The edit distance determination subunit 122 is used to determine the edit distance between the target response content and the preset response template based on the edit distance function;

[0093] A similarity score acquisition subunit 123 is used to combine the semantic similarity with the edit distance using a weighting factor to obtain a similarity score between the target response content and the preset response template, wherein the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template;

[0094] A similarity match failure subunit 124 is used to determine that the target response content fails to match the preset response template if the similarity score is less than a preset score threshold;

[0095] A similarity matching success subunit 125 is used to determine that the target response content matches the preset response template successfully if the similarity score is greater than or equal to a preset score threshold;

[0096] Optionally, the similarity matching success subunit 125 is further used to determine that the target response type of the large language model for the non-safe behavior data is a rejection response when the target response content successfully matches the preset response template;

[0097] The output includes the security assessment results of the target response type of the rejection response.

[0098] An evaluation rule acquisition unit 13 is used to acquire a target evaluation rule corresponding to the non-safe behavior data if the match fails;

[0099] Optionally, the evaluation rule acquisition unit 13 is specifically configured to determine a target evaluation rule corresponding to the non-safe behavior data based on the target risk level if the match fails.

[0100] A prompt word adding unit 14, used to add target evaluation rules, non-safe behavior data and target response content to the target prompt word;

[0101] The security assessment unit 15 is used to perform security assessment on the large language model based on the target prompt word.

[0102] Optionally, the safety assessment unit 15 includes:

[0103] The response type acquisition subunit 151 is used to classify the target response content based on the target evaluation rule, obtain the target response type of the target response content, and determine the target evaluation rule based on which the target response type is determined as the evaluation basis;

[0104] The assessment result output subunit 152 is used to output the security assessment result including the target response type and the assessment basis.

[0105] In an embodiment of the present application, non-safe behavior data is input into a large language model to obtain the target response content of the large language model for the non-safe behavior data, and the non-safe behavior data is classified into a behavior risk level to obtain the target risk level corresponding to the non-safe behavior data. The semantic similarity between the target response content and the preset response template is determined based on a semantic similarity function, and the edit distance between the target response content and the preset response template is determined based on an edit distance function, and then the semantic similarity and the edit distance are combined using a weighting factor to obtain a similarity score between the target response content and the preset response template. If the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully, and the safety assessment result of the model assessment can be directly output; if the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template, and the target assessment rule corresponding to the non-safe behavior data is determined based on the target risk level, and the target assessment rule, non-safe behavior data, and target response content are added to the target prompt word. The target response content is classified based on the target assessment rule to obtain the target response type of the target response content, and the target assessment rule based on which the target response type is determined is determined as the assessment basis, and the safety assessment result including the target response type and the assessment basis is output. The target response content of the large language model is obtained through the non-safe behavior data, so as to perform a series of automatic detections on the target response content, thereby realizing the safety assessment of the large language model, reducing labor costs, and improving the reliability and security of the large language model. The semantic similarity function is combined with the edit distance function using a weighting factor, and then the target response content and the preset response template are matched for similarity, thereby improving the accuracy of similarity matching. Non-safe behavior data is classified into behavioral risk levels to facilitate subsequent determination of target evaluation rules. This eliminates the need to use all evaluation rules for model evaluation, thereby improving the efficiency and accuracy of model evaluation.

[0106] See also Figure 6 , Figure 6 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 6 As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input and output interface module, and a connection establishment application.

[0107] In one embodiment, the processor 1001 may be used to call a connection establishment application stored in the memory 1005, and specifically perform the following operations:

[0108] Inputting non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data;

[0109] Based on the similarity measurement function, the target response content and the preset response template are matched in similarity;

[0110] If the match fails, the target evaluation rules corresponding to the non-safe behavior data are obtained;

[0111] Adding target evaluation rules, non-safe behavior data, and target response content to the target prompt words;

[0112] Security evaluation of large language models based on target cues.

[0113] Optionally, before performing similarity matching between the target response content and the preset response template based on the similarity measurement function, the processor 1001 further performs the following operations:

[0114] Determining a similarity measurement function based on a weighting factor, a semantic similarity function, and an edit distance function;

[0115] The semantic similarity function is used to determine the feature similarity between texts; the edit distance function is used to determine the surface similarity between texts; and the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template.

[0116] Optionally, when the processor 1001 performs similarity matching between the target response content and the preset response template based on the similarity measurement function, the following specific operations are performed:

[0117] Determine the semantic similarity between the target response content and the preset response template based on the semantic similarity function;

[0118] Determine the edit distance between the target response content and the preset response template based on the edit distance function;

[0119] The semantic similarity is combined with the edit distance using a weighting factor to obtain a similarity score between the target response content and the preset response template. The weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template.

[0120] If the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template;

[0121] If the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully.

[0122] Optionally, after executing if the similarity score is greater than or equal to a preset score threshold, determining that the target response content successfully matches the preset response template, the processor 1001 further performs the following operations:

[0123] When the target response content successfully matches the preset response template, the target response type of the large language model for the non-safe behavior data is determined to be a rejection response;

[0124] The output includes the security assessment results of the target response type of the rejection response.

[0125] Optionally, after inputting the non-safe behavior data into the large language model, the processor 1001 further performs the following operations:

[0126] The non-safe behavior data is classified into behavioral risk levels to obtain the target risk level corresponding to the non-safe behavior data.

[0127] Optionally, when the processor 1001 executes the target evaluation rule corresponding to the non-safe behavior data if the match fails, the processor 1001 specifically performs the following operations:

[0128] If the match fails, the target assessment rules corresponding to the non-safe behavior data are determined based on the target risk level.

[0129] Optionally, when performing security assessment on the large language model based on the target prompt word, the processor 1001 specifically performs the following operations:

[0130] Classifying the target response content based on the target evaluation rule to obtain the target response type of the target response content, and determining the target evaluation rule based on which the target response type is determined as the evaluation basis;

[0131] The output includes the target response type and the security assessment results based on the assessment.

[0132] In an embodiment of the present application, non-safe behavior data is input into a large language model to obtain the target response content of the large language model for the non-safe behavior data, and the non-safe behavior data is classified into a behavior risk level to obtain the target risk level corresponding to the non-safe behavior data. The semantic similarity between the target response content and the preset response template is determined based on a semantic similarity function, and the edit distance between the target response content and the preset response template is determined based on an edit distance function, and then the semantic similarity and the edit distance are combined using a weighting factor to obtain a similarity score between the target response content and the preset response template. If the similarity score is greater than or equal to the preset score threshold, it is determined that the target response content matches the preset response template successfully, and the safety assessment result of the model assessment can be directly output; if the similarity score is less than the preset score threshold, it is determined that the target response content fails to match the preset response template, and the target assessment rule corresponding to the non-safe behavior data is determined based on the target risk level, and the target assessment rule, non-safe behavior data, and target response content are added to the target prompt word. The target response content is classified based on the target assessment rule to obtain the target response type of the target response content, and the target assessment rule based on which the target response type is determined is determined as the assessment basis, and the safety assessment result including the target response type and the assessment basis is output. The target response content of the large language model is obtained through the non-safe behavior data, so as to perform a series of automatic detections on the target response content, thereby realizing the safety assessment of the large language model, reducing labor costs, and improving the reliability and security of the large language model. The semantic similarity function is combined with the edit distance function using a weighting factor, and then the target response content and the preset response template are matched for similarity, thereby improving the accuracy of similarity matching. Non-safe behavior data is classified into behavioral risk levels to facilitate subsequent determination of target evaluation rules. This eliminates the need to use all evaluation rules for model evaluation, thereby improving the efficiency and accuracy of model evaluation.

[0133] It should be understood that the device provided in the embodiment of the present application is used to execute the above-mentioned model security assessment method, and thus can achieve the same effect as the above-mentioned implementation method.

[0134] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a computer device, the processing module may be used to control and manage the actions of the computer device. The storage module may be used to support the computer device in executing related program codes, etc.

[0135] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0136] In addition, the device provided in the embodiment of the present application can specifically be a chip, component or module, and the chip may include a connected processor and memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a model security assessment method provided in the above embodiment.

[0137] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a model security assessment method provided by the above-mentioned embodiment.

[0138] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a model security assessment method provided by the above embodiment.

[0139] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0140] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0141] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model security assessment method, characterized in that: The method comprises: Inputting non-safety behavior data into the large language model to obtain target response content of the large language model for the non-safety behavior data; Performing similarity matching between the target response content and a preset response template based on a similarity measurement function; If the match fails, obtaining the target evaluation rule corresponding to the non-safe behavior data; Adding the target evaluation rule, the unsafe behavior data, and the target response content to the target prompt word; A security assessment is performed on the large language model based on the target prompt word.

2. The method according to claim 1, characterized in that Before the similarity matching between the target response content and the preset response template based on the similarity measurement function is performed, the method further includes: Determining a similarity measurement function based on a weighting factor, a semantic similarity function, and an edit distance function; The semantic similarity function is used to determine the feature similarity between texts; the edit distance function is used to determine the surface similarity between texts; and the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template.

3. The method according to claim 2, characterized in that The similarity matching between the target response content and the preset response template based on the similarity measurement function includes: Determining the semantic similarity between the target response content and a preset response template based on the semantic similarity function; Determining an edit distance between the target response content and the preset response template based on the edit distance function; Combining the semantic similarity with the edit distance using the weighting factor to obtain a similarity score between the target response content and the preset response template, wherein the weighting factor is used to balance the surface similarity and feature similarity between the target response content and the preset response template; If the similarity score is less than a preset score threshold, it is determined that the target response content fails to match the preset response template; If the similarity score is greater than or equal to a preset score threshold, it is determined that the target response content successfully matches the preset response template.

4. The method according to claim 3, characterized in that If the similarity score is greater than or equal to the preset score threshold, after determining that the target response content successfully matches the preset response template, the method further includes: When the target response content successfully matches the preset response template, determining that the target response type of the large language model for the non-safe behavior data is a rejection response; Outputting a security assessment result including a target response type of the rejection response.

5. The method according to claim 1, characterized in that After the non-safe behavior data is input into the large language model, the method further includes: The non-safe behavior data is classified into behavior risk levels to obtain a target risk level corresponding to the non-safe behavior data.

6. The method according to claim 5, characterized in that If the match fails, obtaining the target evaluation rule corresponding to the non-safe behavior data includes: If the match fails, the target assessment rule corresponding to the non-safe behavior data is determined based on the target risk level.

7. The method according to claim 1, characterized in that The performing security assessment on the large language model based on the target prompt word includes: Classifying the target response content based on the target evaluation rule to obtain a target response type of the target response content, and determining the target evaluation rule based on which the target response type is determined as an evaluation basis; The output includes the target response type and the security assessment result of the assessment basis.

8. A model security assessment device, characterized in that: The device comprises: a response acquisition unit, configured to input non-safety behavior data into the large language model to acquire target response content of the large language model for the non-safety behavior data; A similarity matching unit, used for performing similarity matching between the target response content and a preset response template based on a similarity measurement function; An evaluation rule acquisition unit, configured to acquire a target evaluation rule corresponding to the non-safe behavior data if the match fails; A prompt word adding unit, used for adding the target evaluation rule, the non-safe behavior data and the target response content to the target prompt word; A security assessment unit is used to perform a security assessment on the large language model based on the target prompt word.

9. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used for page interaction, the memory is used for storing program code, and the processor is used for calling the program code to execute the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images