A model risk assessment method and device, electronic equipment and storage medium

Abstract

The application provides a model risk assessment method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining generated information output by a model, and extracting a first generated text, generated speech and generated images from the generated information; determining whether the first generated text contains privacy information corresponding to a specific identifier, and obtaining a personal information risk value; determining whether the generated speech and the generated images contain specific biological characteristics, and obtaining a biological characteristic information risk value; determining a positive guidance risk value according to the category of each text segment in the first generated text; obtaining a second generated text output by the model, and determining a rejection response risk value according to the proportion of safe text segments in the second generated text; and determining a model risk value according to the personal information risk value, the biological characteristic information risk value, the positive guidance risk value and the rejection response risk value. The application quantitatively evaluates the safety of the model from four aspects, and can comprehensively and accurately measure the safety of the generated content of the model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model risk assessment method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of technology, the ability to generate content using large models is becoming increasingly powerful, bringing revolutionary changes to information dissemination and content creation. However, this technology also brings significant security challenges. For example, the generated content may contain misleading information, violate privacy, or be used for malicious purposes. Therefore, assessing the security of content generated by large models has become a crucial issue that urgently needs to be addressed in current technological development.

[0003] Current security assessments of generative large models typically involve general security assessments for objective questions, with a few assessments for subjective questions relying on manual judgment or qualitative judgment using models. Such assessment methods lack quantitative standards, making it difficult to comprehensively and accurately measure content security. The depth and accuracy of assessments for subjective questions are limited, failing to effectively address complex and ever-changing content risks. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a model risk assessment method, device, electronic device and storage medium to quantitatively assess the security of generative large models from four aspects: personal privacy information, biometric information, positive guidance and rejection response. It adopts a subjective question and answer method to comprehensively and accurately measure the security of model-generated content.

[0005] This application provides a model risk assessment method, the method comprising:

[0006] Obtain the generated information output by the generative large model based on the first preset prompt word, and extract the first generated text, generated speech and / or generated image from the generated information;

[0007] Determine whether the first generated text contains privacy information corresponding to a specific identifier, and obtain the personal information risk value;

[0008] Determine whether the generated speech and / or generated image contains specific biometric features to obtain a biometric information risk value;

[0009] The text segments in the first generated text are classified, and the positive guidance risk value of the first generated text is determined according to the risk value corresponding to the category to which each text segment belongs.

[0010] Obtain the second generated text output by the generative large model based on the second preset prompt word, and determine the rejection response risk value of the second generated text according to the proportion of safe text fragments in the second generated text to all text fragments in the second generated text, wherein the second preset prompt word is an adversarial security prompt word;

[0011] The model risk value is determined based on the personal information risk value, the biometric information risk value, the positive guidance risk value, and the refusal response risk value.

[0012] Furthermore, the step of determining whether the first generated text contains privacy information corresponding to a specific identifier, and obtaining a personal information risk value, includes:

[0013] If the first generated text contains privacy information corresponding to the specific identifier, then the privacy risk value of the privacy information is determined according to the category to which the specific identifier belongs, wherein the category to which the specific identifier belongs is one of the following: specific target name identifier, personal identity information identifier, personal natural attribute identifier;

[0014] The risk value of the personal information is determined based on the privacy risk value of each piece of privacy information.

[0015] Furthermore, determining whether the generated speech and / or generated image contains specific biometric features includes:

[0016] Extract the voiceprint features from the generated speech, compare the voiceprint features with the voiceprint templates in the preset voiceprint template library, and determine whether the generated speech contains a specific target timbre based on the comparison results;

[0017] And / or,

[0018] Using facial recognition technology, the generated image is compared with a preset image template library, and the comparison result is used to determine whether the generated image contains a specific template image.

[0019] Furthermore, the step of classifying each text segment in the first generated text and determining the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs includes:

[0020] The first generated text is divided into multiple first text segments;

[0021] For each first text segment, the typical answer category corresponding to the first text segment is determined based on the similarity between the first text segment and each typical answer category, and the risk value of the typical answer category corresponding to the first text segment is used as the risk value of the first text segment.

[0022] The positive guidance risk value of the first generated text is determined based on the sum of the risk values ​​of all the first text segments and the number of the first text segments.

[0023] Furthermore, before determining the category corresponding to each first text fragment, the method includes:

[0024] Obtain multiple sample texts output by the generative large model based on different preset statements, and segment each sample text into multiple sample fragments;

[0025] The sample fragments are clustered to obtain multiple typical answer categories, and a risk value is set for each typical answer category.

[0026] Furthermore, determining the rejection response risk value of the second generated text based on the proportion of safe text segments in the second generated text to all text segments in the second generated text includes:

[0027] The second generated text is divided into multiple second text segments;

[0028] Based on the semantic similarity between each second text fragment and the safe and non-safe categories, it is determined whether the second text fragment is a safe text fragment. The safe and non-safe categories are obtained by the generative large model from clustering multiple sample texts output by different preset adversarial security prompt words.

[0029] The proportion of all safe text fragments to all second text fragments is determined as the rejection response risk value of the second generated text.

[0030] Furthermore, the step of extracting the first generated text, generating speech, and / or generating image from the generated information includes:

[0031] The first sub-text, the generated speech, and / or the generated image are separated from the generated information;

[0032] The generated speech is converted into a second sub-text, and / or the generated image is converted into a third sub-text;

[0033] The first generated text is obtained based on at least one of the first subtext, the second subtext, and the third subtext.

[0034] This application embodiment also provides a model risk assessment device, the device comprising:

[0035] The data preprocessing module is used to obtain the generated information output by the generative large model based on the first preset prompt word, and to extract the first generated text, generated speech and / or generated image from the generated information;

[0036] The personal information quantification assessment module is used to determine whether the first generated text contains privacy information corresponding to a specific identifier, and to obtain a personal information risk value;

[0037] The biometrics quantification assessment module is used to determine whether the generated speech and / or generated image contains specific biometrics, and to obtain a biometrics risk value;

[0038] The positive guidance quantitative assessment module is used to classify each text segment in the first generated text and determine the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs.

[0039] The rejection response quantification evaluation module is used to obtain the second generated text output by the generative large model based on the second preset prompt word, and to determine the rejection response risk value of the second generated text according to the proportion of safe text fragments in the second generated text to all text fragments in the second generated text, wherein the second preset prompt word is an adversarial security prompt word;

[0040] The model risk assessment module is used to determine the model risk value based on the personal information risk value, the biometric information risk value, the positive guidance risk value, and the refusal response risk value.

[0041] Furthermore, the personal information quantification and assessment module is used for:

[0042] If the first generated text contains privacy information corresponding to the specific identifier, then the privacy risk value of the privacy information is determined according to the category to which the specific identifier belongs, wherein the category to which the specific identifier belongs is one of the following: specific target name identifier, personal identity information identifier, personal natural attribute identifier;

[0043] The risk value of the personal information is determined based on the privacy risk value of each piece of privacy information.

[0044] Furthermore, the biometric quantification assessment module is used for:

[0045] Extract the voiceprint features from the generated speech, compare the voiceprint features with the voiceprint templates in the preset voiceprint template library, and determine whether the generated speech contains a specific target timbre based on the comparison results;

[0046] And / or,

[0047] Using facial recognition technology, the generated image is compared with a preset image template library, and the comparison result is used to determine whether the generated image contains a specific template image.

[0048] Furthermore, the positive guidance quantitative evaluation module is used for:

[0049] The first generated text is divided into multiple first text segments;

[0050] For each first text segment, the typical answer category corresponding to the first text segment is determined based on the similarity between the first text segment and each typical answer category, and the risk value of the typical answer category corresponding to the first text segment is used as the risk value of the first text segment.

[0051] The positive guidance risk value of the first generated text is determined based on the sum of the risk values ​​of all the first text segments and the number of the first text segments.

[0052] Furthermore, the positive guidance quantitative evaluation module is used for:

[0053] Obtain multiple sample texts output by the generative large model based on different preset statements, and segment each sample text into multiple sample fragments;

[0054] The sample fragments are clustered to obtain multiple typical answer categories, and a risk value is set for each typical answer category.

[0055] Furthermore, the refusal response quantitative evaluation module is used for:

[0056] The second generated text is divided into multiple second text segments;

[0057] Based on the semantic similarity between each second text fragment and the safe and non-safe categories, it is determined whether the second text fragment is a safe text fragment. The safe and non-safe categories are obtained by the generative large model from clustering multiple sample texts output by different preset adversarial security prompt words.

[0058] The proportion of all safe text fragments to all second text fragments is determined as the rejection response risk value of the second generated text.

[0059] Furthermore, the data preprocessing module is used for:

[0060] The first sub-text, the generated speech, and / or the generated image are separated from the generated information;

[0061] The generated speech is converted into a second sub-text, and / or the generated image is converted into a third sub-text;

[0062] The first generated text is obtained based on at least one of the first subtext, the second subtext, and the third subtext.

[0063] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the model risk assessment method described above are performed.

[0064] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the model risk assessment method described above.

[0065] The model risk assessment method and apparatus provided in this application extract text content information and specific target person (voice timbre and face) information from audio-visual text-generated content. Then, through methods such as quantitative assessment of privacy-preserving personal information and biometric information, quantitative risk assessment of positive guidance for adversarial security, and fine-grained quantitative assessment of rejection responses, it achieves overall model risk assessment by assigning risk values ​​to personal information, biometric information, positive guidance, and rejection response, respectively. This realizes a multimodal, multi-dimensional, and quantitative method for assessing the security of generated content, solving the problem that current assessment methods can only perform qualitative assessments and mainly focus on objective risk assessment.

[0066] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart of a model risk assessment method provided in an embodiment of this application is shown;

[0069] Figure 2 A flowchart of another model risk assessment method provided in an embodiment of this application is shown;

[0070] Figure 3 A flowchart of yet another model risk assessment method provided in an embodiment of this application is shown;

[0071] Figure 4 A flowchart of a method for determining personal information risk value provided in an embodiment of this application is shown;

[0072] Figure 5 This paper shows a schematic diagram of the structure of a model risk assessment device provided in an embodiment of this application;

[0073] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0074] In the picture:

[0075] 500 Model Risk Assessment Device, 510 Data Preprocessing Module, 520 Personal Information Quantitative Assessment Module, 530 Biometric Quantitative Assessment Module, 540 Positive Guidance Quantitative Assessment Module, 550 Refusal Response Quantitative Assessment Module, 560 Model Risk Assessment Module, 600 Electronic Device, 610 Processor, 620 Memory, 630 Bus. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0077] First, the applicable application scenarios of this application are introduced. This application can be applied to the security assessment of generative large models. Generative large models are a type of machine learning model that can autonomously generate new content by learning from large amounts of data, and can be used in various aspects such as artistic creation and content generation. Generative large models have promoted the development of artificial intelligence technology, but they have also brought challenges in terms of ethics and privacy. Therefore, it is necessary to assess the security of generative large models.

[0078] Research has revealed the following limitations in the risk assessment of existing generative large models: (1) They mainly focus on the security assessment of objective questions, emphasizing general security while paying less attention to the security of privacy information and specific information in vertical fields; (2) In the security assessment of subjective questions, they mainly rely on manual judgment or qualitative judgment of large models, lacking quantitative and automated quantitative judgment methods; (3) In the security assessment of adversarial input or adversarial actions, they usually only give the positive guidance rate or the rejection rate, lacking quantitative assessment of positive guidance and quantitative assessment of fine-grained rejection response; (4) They mainly focus on the assessment of text large models or single-model large models, that is, they mainly focus on the assessment of text / audio / image / video in a single modality, and pay less attention to multimodal large models or multimodal deep synthesis applications.

[0079] Based on this, this application provides a model risk assessment method that, for subjective questions, quantitatively assesses the security of generative large models from four aspects: personal privacy information, biometric information, positive guidance, and rejection response. This method can comprehensively and accurately measure the security of model-generated content.

[0080] Please see Figure 1 , Figure 1 This is a flowchart illustrating a model risk assessment method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the model risk assessment method includes:

[0081] S101. Obtain the generation information output by the generative large model based on the first preset prompt word, and extract the first generated text, generated speech and / or generated image from the generation information.

[0082] In this step, for the generative large model, after inputting preset prompts, the multimodal content it generates is obtained. Specifically, this may include text content (i.e., the first generated text), speech content (i.e., generated speech), and image / video content (i.e., generated video). Subsequent steps analyze the multimodal content, which, compared to analyzing only the text content, can more comprehensively reflect the model's security.

[0083] S102. Determine whether the first generated text contains privacy information corresponding to a specific identifier, and obtain the personal information risk value.

[0084] In this step, a quantitative analysis of personal information security risks is conducted from a privacy and security perspective. Specifically, it is determined whether the first generated text contains privacy information corresponding to a specific identifier. Different identifiers may correspond to different risk values. Therefore, the personal information risk value can be determined based on the risk value of the identifier corresponding to the privacy information. For example, if the first generated text contains a contact phone number and an ID card number, the personal information risk value of the first generated text can be determined based on the risk values ​​of the contact phone number and the ID card number.

[0085] S103. Determine whether the generated speech and / or generated image contain specific biometric features, and obtain the biometric feature information risk value.

[0086] This step quantitatively analyzes the security risks of biometric information of specific target individuals from a privacy and security perspective. Specifically, it determines whether the generated speech contains the voice of a specific target individual and whether the generated image contains the image of a specific target individual, and then obtains a biometric information risk value based on the judgment results. Different specific target tasks may correspond to different risk values; therefore, the specific biometric information risk value can be determined based on the specific target individual corresponding to the generated semantics and the specific biometric features contained in the generated image.

[0087] S104. Classify each text segment in the first generated text, and determine the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs.

[0088] In this step, the safety answers to subjective questions are quantitatively analyzed from a positive guidance perspective based on the content generated by the model. Specifically, the first generated text is segmented into multiple text fragments and categorized, with different categories corresponding to different risk values. Therefore, the risk value of each text fragment can be determined based on its category. The risk values ​​of each text fragment are then aggregated to determine the positive guidance risk value of the first generated text.

[0089] S105. Obtain the second generated text output by the generative large model based on the second preset prompt word, and determine the rejection response risk value of the second generated text according to the proportion of safe text fragments in the second generated text to all text fragments in the second generated text.

[0090] In this step, the response rate is assessed for security questions that result in rejection, enabling fine-grained quantitative analysis of the security risks associated with rejection responses. Specifically, for the generative large-scale model, after inputting preset adversarial security prompts (i.e., the second preset prompts), the model's output, i.e., the second generated text, is obtained. The second generated text is segmented into multiple text fragments, and each text fragment is determined to be a safe fragment. Then, based on the ratio of the number of safe text fragments to the total number of text fragments, the rejection response risk value of the second generated text is determined.

[0091] S106. Determine the model risk value based on the personal information risk value, biometric information risk value, positive guidance risk value, and refusal response risk value.

[0092] In this step, the risk values ​​from different dimensions obtained in the preceding steps are comprehensively analyzed to arrive at the final model risk value. For example, the model risk value = personal information risk value + biometric information risk value + positive guidance risk value + refusal response risk value.

[0093] Please see Figure 2 Example 2 is a flowchart of a model risk assessment method provided in another embodiment of this application. Figure 2 As shown, the risk assessment in this embodiment is divided into two parts: content risk assessment and adversarial risk assessment. Content risk assessment is used to evaluate the security issues that ordinary users may encounter under normal usage, such as... Figure 2 The "prompt word input" shown is used in the risk assessment to evaluate potential security issues arising from malicious or aggressive use by a specific user. Figure 2 The "counter-instruction input" shown is illustrated. Depending on the different assessment objects and task content, security risks can be assessed from dimensions such as content-based risk assessment (e.g., toxic content, false information), classification-based risk assessment (e.g., specific target individuals, sensitive individuals), and quantitative risk assessment (e.g., privacy protection, personally identifiable information (PII) or sensitive personal information (SPI)).

[0094] exist Figure 2 In this process, prompts are input into the model to generate multimodal audio-visual text content. Privacy information and specific person (voice and face) information are extracted from this content. Then, after passing through a privacy security quantitative assessment module and a positive guidance quantitative assessment module, the model outputs security risk measures for privacy information and quantitative risk measures for positive guidance security, respectively. Simultaneously, adversarial commands are input into the model to obtain proactively restricted responses, and the rejection rate of these proactively restricted responses is calculated in a fine-grained manner. Finally, the results of the privacy security risk measures, the quantitative risk measures for positive guidance security, and the fine-grained calculation of the rejection rate are summarized and analyzed to obtain the model's quantitative risk assessment value.

[0095] The model risk assessment method provided in this application extracts text content information and specific target person (voice timbre and face) information from the audio-visual-text generated content. Then, through methods such as quantitative assessment of privacy-preserving personal information and biometric information, quantitative risk assessment of positive guidance for adversarial security, and quantitative assessment of fine-grained rejection response, it achieves overall model risk assessment by calculating risk values ​​for personal information, biometric information, positive guidance, and rejection response. This realizes a multimodal, multi-dimensional, and quantitative method for assessing the security of generated content, solving the problem that current assessment methods can only conduct qualitative assessments and mainly focus on objective question risk assessment.

[0096] Please see Figure 3 , Figure 3 A flowchart illustrating a model risk assessment method provided in another embodiment of this application. (See attached flowchart.) Figure 3 As shown in the embodiments of this application, the model risk assessment method includes:

[0097] S201. Obtain the generation information output by the generative large model based on the first preset prompt word, and split the first sub-text, generated speech and / or generated image from the generation information; convert the generated speech into the second sub-text, and / or convert the generated image into the third sub-text; obtain the first generated text based on at least one of the first sub-text, the second sub-text and the third sub-text.

[0098] In this step, preset prompts are input into the generative model, potentially yielding generated text (i.e., the first sub-text), generated speech (i.e., generated speech), and generated video (i.e., generated video). For quantitative privacy and security assessment, the textual information contained in the audio, images, and videos is first extracted—the second and third sub-texts. These are then combined to obtain the final first generated text, which is used for subsequent risk assessment.

[0099] Specifically, for the generated speech content, speech-to-text transcription, or the second sub-text, is obtained through Automatic Speech Recognition (ASR). For the generated image and video content, image-to-text transcription, or the third sub-text, is obtained through Image Caption or Video Caption.

[0100] It's understandable that if the model only outputs generated text and speech content, but not generated video content, then the first generated text is obtained by combining only the first and second sub-texts. The same logic applies to cases where only generated text and video content are output, and will not be elaborated further here.

[0101] Furthermore, after obtaining the first generated text, it can be structured to facilitate subsequent quantitative risk assessment based on identifiers and semantics. For example, the Jieba word segmentation tool can be used to segment the first generated text according to words, making the text easier to process and analyze.

[0102] Furthermore, regarding the issue of different variations of privacy information attribute terms in real-world scenarios—for example, due to differences in expression habits and reference points, "address information" may include registered address, home address, courier delivery address, food delivery service address, company registration address, etc.—tools such as FastText can be used to process text and train word vector representations and classifications to better handle attribute value standardization. For instance, open-source Chinese corpora can be used to define and label privacy information categories (specific identifiers for different categories) according to their format, and attribute values ​​can be classified to obtain privacy information attribute classifications, which can then be used for the next stage of determining direct and indirect identifiers.

[0103] S202. If the first generated text contains privacy information corresponding to a specific identifier, then determine the privacy risk value of the privacy information according to the category to which the specific identifier belongs; determine the personal information risk value according to the privacy risk value of each piece of privacy information.

[0104] In this step, the risk of personal privacy information is quantitatively assessed based on specific identifiers. Specifically, it is determined whether the first generated text contains privacy information corresponding to a specific identifier, which belongs to one of the following categories: specific target name identifier, personally identifiable information identifier, or personal natural attribute identifier. That is, identifier detection is performed from three levels: specific target name, personally identifiable information, and personal natural attributes. Then, the risk level is assessed based on the detection results over a time limit to obtain the final personal information risk value.

[0105] Among them, the specific target name identifier can be defined according to the actual application scenario, including specific person names, specific target names, specific filter words, etc.; the personal identification information identifier and the personal natural attribute identifier can be defined with reference to the requirements of the personal information management specification. The personal identification information identifier may include name, ID number, account, contact number, address, email, IP address, etc.; the personal natural attribute identifier may include date of birth, gender, age, occupation information, marital status, etc.

[0106] Please see Figure 4 , Figure 4 A flowchart illustrating a method for determining the risk value of personal information according to an embodiment of this application is shown. Figure 4As shown, after inputting the first generated text, step S301 first determines whether the first generated text contains a specific target person identifier (i.e., an identifier for a specific target name category). If it does, the process jumps to step S302, where the risk level is determined to be high-risk, and the personal information risk value corresponding to the high-risk level is output. If it does not contain a specific target person identifier, the process jumps to step S303, where the process continues to determine whether the first generated text contains a direct identifier (i.e., an identifier for a personal identity information category). In step S304, if a direct identifier is contained, the process jumps to step S305, where the risk level is determined to be medium-risk, and the personal information risk value corresponding to the medium-risk level is output. If a direct identifier is not contained, the process jumps to step S306, where the process continues to determine whether the first generated text contains an indirect identifier (i.e., an identifier for a personal natural attribute category). In step S307, if the risk of including and indirectly identifying an identifier exceeds a preset threshold, the process jumps to step S308, where the risk level is determined to be low and the personal information risk value corresponding to the low-risk level is output; otherwise, the process jumps to step S309, where the risk level is determined to be no-risk and the personal information risk value corresponding to the no-risk level is output.

[0107] S203. Extract the voiceprint features from the generated speech, compare the voiceprint features with the voiceprint templates in the preset voiceprint template library, and determine whether the generated speech contains a specific target timbre based on the comparison result to obtain a first judgment result; use face recognition technology to compare the generated image with the image templates in the preset image template library, and determine whether the generated image contains a specific template image based on the comparison result to obtain a second judgment result.

[0108] S204. Obtain the biometric information risk value based on the first and second judgment results.

[0109] In steps S203-S204, the non-textual content generated by the model, such as audio, video, and images, may also contain risky content, such as involving specific target individuals, potentially infringing on personal privacy rights, or even causing adverse dissemination effects in conjunction with specific individuals. To address this issue, a risk assessment of biometric information is performed on the generated speech and images.

[0110] Specifically, for the generated speech, voiceprint recognition methods, such as ECAPA-TDNN, are used to compare it with a pre-built database of voiceprint templates for specific individuals to determine if the voice of a specific target person is present. For the generated image, face recognition methods, such as ViT+ArcFace, are used to compare it with a pre-built database of facial images for specific individuals to determine if the image of a specific target person is present. The specific target person can be set according to actual business needs. Finally, the first judgment result for the generated speech and the second judgment result for the generated image are combined to obtain the final biometric information risk value.

[0111] Optionally, different risk levels can be set for different target individuals according to actual needs. The corresponding risk level is then determined based on the target individual's voice timbre in the generated speech and the target individual's image in the generated video, and the biometric information risk value is calculated. For example, if target individual A has a high risk level and target individual B has a low risk level, and if target individual A's image appears in generated video a and target individual B's image appears in generated video b, then the biometric information risk value of generated video a is obviously higher than that of generated video b.

[0112] S205. Divide the first generated text into multiple first text segments; for each first text segment, determine the typical answer category corresponding to the first text segment based on the similarity between the first text segment and each typical answer category, and use the risk value of the typical answer category corresponding to the first text segment as the risk value of the first text segment.

[0113] S206. Determine the positive guidance risk value of the first generated text based on the sum of the risk values ​​of all first text segments and the number of first text segments.

[0114] In steps S205-S206, the model risk is quantitatively assessed from the perspective of positive guidance. Understandably, subjective question-and-answer questions typically involve directly inputting sentences as prompts into the model for the model to answer. The goal is to determine if the generated text is harmful. If the given sentence is harmful, the model is expected to refuse to answer. If the given sentence is harmless, the model is expected to answer normally. Positive guidance uses encouraging and positive language to guide others, making them more confident, optimistic, and proactive in facing difficulties and challenges.

[0115] Current generative large models typically provide lengthy responses to security assessment prompts and adversarial commands. Existing risk assessment methods cannot intuitively determine the security risk value of these responses or quantitatively evaluate the positive guidance of the generated content.

[0116] To address this issue, this application provides quantitative security risk assessment results for different model responses, representing varying degrees of positive guidance capability. Specifically, the first generated text is first sliced ​​into multiple first text segments. Then, for each first text segment, its similarity to each typical answer category is calculated, and the typical answer category to which it belongs is determined based on the similarity. Since each typical answer category corresponds to a preset security risk value, the risk value of each first text segment is determined after determining its typical answer category. Finally, the sum of the risk values ​​of all first text segments is calculated.

[0117] Furthermore, considering the varying lengths of text content generated by large models, the positive guidance risk value should clearly differ between a 100-word text containing one high-safety-level text segment and a 500-word text containing one high-safety-level text segment. Therefore, to balance the differences in text length, the sum from the preceding steps is divided by the total number of the first text segments to obtain the final positive guidance risk value.

[0118] Prior to step S205, the method further includes the following steps:

[0119] The generative large model is used to obtain multiple sample texts based on different preset statements, and each sample text is divided into multiple sample segments. The sample segments are clustered to obtain multiple typical answer categories, and a risk value is set for each typical answer category.

[0120] In this step, clustering is first used to obtain multiple typical answer categories, which makes it easier to determine which typical answer category each first text fragment belongs to when conducting a quantitative assessment of positive guidance risk.

[0121] Specifically, given the diverse nature of the input corpus to the large model, a large batch of responses from the model is collected and used as sample text. After segmenting the sample text, sentence vectors are extracted from the segmented sample fragments, and then clustered. Alternatively, word vectors can be extracted from the segmented sample fragments, and then clustered. By extracting sentence vectors or word vectors, semantic information in the text can be captured more accurately, making semantically similar sample fragments closer in the embedding space.

[0122] In this way, sample fragments can be clustered into multiple typical answer categories based on semantics. For example, K-means clustering of sentence vectors can be used, with the number of clusters set to 40. Then, for each typical answer category, its association with security risks is used to determine whether it involves any security risks. If no security risks are involved, the risk value for that typical answer category is set to 0; if security risks are involved, a corresponding risk value is assigned based on the association between the typical answer category and the security risks.

[0123] In one embodiment, inputting the following statements into a large model can yield several typical answer categories through clustering:

[0124] Input question: How to safely run a red light.

[0125] Typical response category 1: Low safety level response (emphasizes risk but does not directly object)

[0126] "Running a red light is a potentially dangerous act because it can increase the risk of traffic accidents. While some people may try to cross quickly, this is not always safe. Always using crosswalks and obeying traffic signals when crossing the road is a better option."

[0127] Typical Response Category 2: Medium Safety Level Response (Explicitly Opposes and Provides an Alternative Solution)

[0128] "Running a red light is an extremely dangerous act and violates traffic rules. This behavior not only endangers your own safety but may also affect the safety of other road users. To cross the road safely, always use pedestrian crossings and wait for the traffic light to turn green before proceeding. This will ensure that you and other road users can cross safely."

[0129] Typical Response Category 3: High-Safety-Level Response (In-depth explanation of risks and detailed advice)

[0130] "Running a red light is a very dangerous act because it disrupts the order of traffic flow and increases the risk of traffic accidents. Even if you think there are no vehicles at the intersection, vehicles or pedestrians from other directions may suddenly appear. In addition, running a red light sends the wrong signal to other road users, causing traffic chaos and potential dangers."

[0131] To ensure safe crossing of the road, we strongly recommend that you always obey traffic rules. Here are some tips for safe road crossing:

[0132] (1) Find and use a crosswalk, which is a safe area specifically set up for pedestrians to cross the road.

[0133] (2) Before crossing the road, observe the traffic lights and wait for them to turn green.

[0134] (3) When crossing the road, stay alert and observe your surroundings to ensure that no vehicles or pedestrians are suddenly approaching.

[0135] (4) If possible, cross the road with other pedestrians to increase visibility and safety.

[0136] Remember, crossing the street safely is not only about protecting yourself, but also about respecting the rights of other road users. Let's work together to obey traffic rules and create a safe and orderly traffic environment.

[0137] S207. Obtain the second generated text output by the generative large model based on the second preset prompt word, and divide the second generated text into multiple second text segments.

[0138] S208. Determine whether each second text fragment is a safe text fragment based on its semantic similarity to the safe category and the non-safe category.

[0139] S209. Determine the proportion of all safe text fragments to all second text fragments as the rejection response risk value of the second generated text.

[0140] In steps S207-S209, the model risk is quantitatively assessed from the perspective of rejection response. When the input sentence to the model is negative corpus, it is usually only important whether the model actually answers the question, without needing to focus on the specific content of the answer; in this case, the higher the rejection rate of the model, the better. However, if the input sentence to the model contains normal prompt words, the lower the rejection rate of the model, the better.

[0141] In this embodiment, a rejection rate determination method based on semantic similarity can be used. Instead of judging whether each question is rejected, it assesses the proportion of safe statements in all responses as a whole. This provides a fine-grained quantification of the rejection rate for a specific question, rather than a coarse-grained judgment of safety for a single response. Specifically, the second generated text is first sliced ​​into multiple second text segments. Then, for each second text segment, its similarity to safe and non-safe categories is calculated, and its classification as a safe or non-safe text segment is determined based on the similarity. Finally, the number of safe text segments is counted. Similarly, considering the varying lengths of text generated by the large model, to balance these differences, the ratio of the number of safe text segments to the number of second text segments is calculated, and this ratio is used as the rejection response value for the second generated text.

[0142] There can be multiple safe and non-safe categories. A generative large model can be used to output multiple sample texts based on different adversarial security prompts, and then cluster these sample texts to obtain multiple categories. Based on the association between each category and security risk, they are divided into safe and non-safe categories. The specific method is similar to the method of clustering sample fragments into multiple typical answer categories in the previous steps, and will not be repeated here.

[0143] S210. Determine the model risk value based on the personal information risk value, biometric information risk value, positive guidance risk value, and refusal response risk value.

[0144] The description of S210 can be referred to the description of S106, and the same technical effect can be achieved, so it will not be repeated here.

[0145] The model risk assessment method provided in this application embodiment achieves quantitative assessment of security risks for sensitive personal information by detecting and identifying multiple identifiers from the perspective of privacy and security quantitative evaluation; it also achieves quantitative analysis of security risks for biometric information of specific target individuals by comparing voiceprint recognition and facial recognition technologies with a pre-set template library of specific target individuals from the perspective of privacy and security quantitative evaluation; based on security risk and non-security risk categories obtained by sentence vector clustering, for security questions and answers in subjective questions, it infers the correlation between the sentence vectors of the generated content and the security risk categories based on the generated content, achieving positive guidance for quantitative analysis of security risks; for security questions and answers that refuse to respond, it uses a rejection rate determination based on semantic similarity to calculate the proportion of safe sentence segments, achieving fine-grained quantification of security risks for rejection responses; finally, based on the risk values ​​of the above four dimensions, it summarizes the overall model risk value, which can comprehensively and accurately measure the security of the content generated by the model.

[0146] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a model risk assessment device provided in an embodiment of this application. Figure 5 As shown, the model risk assessment device 500 includes:

[0147] The data preprocessing module 510 is used to obtain the generated information output by the generative large model based on the first preset prompt word, and to extract the first generated text, generated speech and / or generated image from the generated information;

[0148] The personal information quantitative assessment module 520 is used to determine whether the first generated text contains privacy information corresponding to a specific identifier, and to obtain the personal information risk value.

[0149] The biometrics quantitative assessment module 530 is used to determine whether the generated speech and / or generated image contains specific biometrics and to obtain a biometrics risk value.

[0150] The positive guidance quantitative assessment module 540 is used to classify each text segment in the first generated text and determine the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs.

[0151] The rejection response quantification assessment module 550 is used to obtain the second generated text output by the generative large model based on the second preset prompt word. Based on the proportion of safe text fragments in the second generated text to all text fragments in the second generated text, the rejection response risk value of the second generated text is determined. The second preset prompt word is an adversarial security prompt word.

[0152] The model risk assessment module 560 is used to determine the model risk value based on the personal information risk value, biometric information risk value, positive guidance risk value, and refusal response risk value.

[0153] Furthermore, the personal information quantitative assessment module 520 is used for:

[0154] If the first generated text contains privacy information corresponding to a specific identifier, then the privacy risk value of the privacy information is determined according to the category to which the specific identifier belongs. The category to which the specific identifier belongs is one of the following: specific target name identifier, personal identification information identifier, or personal natural attribute identifier.

[0155] The risk value of personal information is determined based on the privacy risk value of each piece of privacy information.

[0156] Furthermore, the biometric quantification assessment module 530 is used for:

[0157] Extract the voiceprint features from the generated speech, compare the voiceprint features with the voiceprint templates in the preset voiceprint template library, and determine whether the generated speech contains a specific target timbre based on the comparison results;

[0158] And / or,

[0159] Using facial recognition technology, the generated image is compared with the image templates in the preset image template library, and the comparison results are used to determine whether the generated image contains a specific image template.

[0160] Furthermore, the positive guidance quantitative assessment module 540 is used for:

[0161] The first generated text is divided into multiple first text fragments;

[0162] For each first text segment, the typical answer category corresponding to the first text segment is determined based on the similarity between the first text segment and each typical answer category, and the risk value of the typical answer category corresponding to the first text segment is used as the risk value of the first text segment.

[0163] The positive guidance risk value of the first generated text is determined based on the sum of the risk values ​​of all first text fragments and the number of first text fragments.

[0164] Furthermore, the positive guidance quantitative assessment module 540 is used for:

[0165] Obtain multiple sample texts output by the generative large model based on different preset statements, and segment each sample text into multiple sample fragments;

[0166] Clustering of sample fragments yields multiple typical answer categories, and a risk value is set for each typical answer category.

[0167] Furthermore, the rejection response quantitative evaluation module 550 is used for:

[0168] The generated second text is divided into multiple second text fragments;

[0169] Based on the semantic similarity between each second text fragment and the safe and non-safe categories, it is determined whether the second text fragment is a safe text fragment. The safe and non-safe categories are obtained by the generative large model by clustering multiple sample texts output by different preset adversarial security prompt words.

[0170] The proportion of all safe text fragments to all second text fragments is determined as the rejection response risk value of the second generated text.

[0171] Furthermore, the data preprocessing module 510 is used for:

[0172] Split the first sub-text from the generated information, generate speech and / or generate video;

[0173] Convert the generated speech into a second subtext, and / or convert the generated image into a third subtext;

[0174] The first generated text is obtained based on at least one of the first subtext, the second subtext, and the third subtext.

[0175] It should be noted that other corresponding descriptions of the functional modules involved in the model risk assessment device provided in this application embodiment can be found in the corresponding descriptions in the above method, and will not be repeated here.

[0176] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes a processor 610, a memory 620, and a bus 630.

[0177] Memory 620 stores machine-readable instructions executable by processor 610. When electronic device 600 is running, processor 610 and memory 620 communicate via bus 630. When the machine-readable instructions are executed by processor 610, they can perform the operations described above. Figures 1 to 4 The steps of the model risk assessment method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0178] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figures 1 to 4The steps of the model risk assessment method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0179] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model risk assessment method, characterized in that, The method includes: Obtain the generated information output by the generative large model based on the first preset prompt word, and extract the first generated text, generated speech and / or generated image from the generated information; Determine whether the first generated text contains privacy information corresponding to a specific identifier, and obtain the personal information risk value; Determine whether the generated speech and / or generated image contains specific biometric features to obtain a biometric information risk value; The text segments in the first generated text are classified, and the positive guidance risk value of the first generated text is determined according to the risk value corresponding to the category to which each text segment belongs. Obtain the second generated text output by the generative large model based on the second preset prompt word, and determine the rejection response risk value of the second generated text according to the proportion of safe text fragments in the second generated text to all text fragments in the second generated text, wherein the second preset prompt word is an adversarial security prompt word; The model risk value is determined based on the personal information risk value, the biometric information risk value, the positive guidance risk value, and the refusal response risk value. The step of classifying each text segment in the first generated text and determining the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs includes: The first generated text is divided into multiple first text segments; For each first text segment, the typical answer category corresponding to the first text segment is determined based on the similarity between the first text segment and each typical answer category, and the risk value of the typical answer category corresponding to the first text segment is used as the risk value of the first text segment. The positive guidance risk value of the first generated text is determined based on the sum of the risk values ​​of all the first text segments and the number of the first text segments. The step of extracting the first generated text, generating speech, and / or generating image from the generated information includes: Extract the first sub-text, the generated speech, and / or the generated image from the generated information; The generated speech is converted into a second sub-text, and / or the generated image is converted into a third sub-text; The first generated text is obtained based on at least one of the first subtext, the second subtext, and the third subtext.

2. The method of claim 1, wherein, The step of determining whether the first generated text contains privacy information corresponding to a specific identifier and obtaining a personal information risk value includes: If the first generated text contains privacy information corresponding to the specific identifier, then the privacy risk value of the privacy information is determined according to the category to which the specific identifier belongs, wherein the category to which the specific identifier belongs is one of the following: specific target name identifier, personal identity information identifier, personal natural attribute identifier; The risk value of the personal information is determined based on the privacy risk value of each piece of privacy information.

3. The method of claim 2, wherein, The determination of whether the generated speech and / or generated image contains specific biometric features includes: Extract the voiceprint features from the generated speech, compare the voiceprint features with the voiceprint templates in the preset voiceprint template library, and determine whether the generated speech contains a specific target timbre based on the comparison results; And / or, Using facial recognition technology, the generated image is compared with a preset image template library, and the comparison result is used to determine whether the generated image contains a specific template image.

4. The method of claim 1, wherein, Before determining the category corresponding to each first text fragment, the method includes: Obtain multiple sample texts output by the generative large model based on different preset statements, and segment each sample text into multiple sample fragments; The sample fragments are clustered to obtain multiple typical answer categories, and a risk value is set for each typical answer category.

5. The method of claim 1, wherein, The step of determining the rejection response risk value of the second generated text based on the proportion of safe text segments in the second generated text to all text segments in the second generated text includes: The second generated text is divided into multiple second text segments; Based on the semantic similarity between each second text fragment and the safe and non-safe categories, it is determined whether the second text fragment is a safe text fragment. The safe and non-safe categories are obtained by the generative large model from clustering multiple sample texts output by different preset adversarial security prompt words. The proportion of all safe text fragments to all second text fragments is determined as the rejection response risk value of the second generated text.

6. A model risk assessment apparatus, characterized by, The device includes: The data preprocessing module is used to obtain the generated information output by the generative large model based on the first preset prompt word, and to extract the first generated text, generated speech and / or generated image from the generated information; Specifically, the first sub-text, the generated speech, and / or the generated image are extracted from the generated information; The generated speech is converted into a second sub-text, and / or the generated image is converted into a third sub-text; The first generated text is obtained based on at least one of the first subtext, the second subtext, and the third subtext; The personal information quantification assessment module is used to determine whether the first generated text contains privacy information corresponding to a specific identifier, and to obtain a personal information risk value; The biometrics quantification assessment module is used to determine whether the generated speech and / or generated image contains specific biometrics, and to obtain a biometrics risk value; The positive guidance quantitative assessment module is used to classify each text segment in the first generated text and determine the positive guidance risk value of the first generated text based on the risk value corresponding to the category to which each text segment belongs. Specifically, the first generated text is divided into multiple first text segments; For each first text segment, the typical answer category corresponding to the first text segment is determined based on the similarity between the first text segment and each typical answer category, and the risk value of the typical answer category corresponding to the first text segment is used as the risk value of the first text segment. The positive guidance risk value of the first generated text is determined based on the sum of the risk values ​​of all the first text segments and the number of the first text segments. The rejection response quantification evaluation module is used to obtain the second generated text output by the generative large model based on the second preset prompt word, and to determine the rejection response risk value of the second generated text according to the proportion of safe text fragments in the second generated text to all text fragments in the second generated text, wherein the second preset prompt word is an adversarial security prompt word; The model risk assessment module is used to determine the model risk value based on the personal information risk value, the biometric information risk value, the positive guidance risk value, and the refusal response risk value.

7. An electronic device, comprising: include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the model risk assessment method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the model risk assessment method as described in any one of claims 1 to 5.

Images