Security evaluation method and security evaluation device for large model and electronic equipment

Abstract

The invention relates to the technical field of large model security assessment, and discloses a security assessment method and a security assessment device for a large model, and electronic equipment. The security assessment method comprises the steps of obtaining test text image pairs of multiple fields; wherein the test text image pair comprises a test text and a test image; determining an attack text according to the description information of the role corresponding to each field and the test text, and determining an attack image according to the test image and the attack text corresponding to the test image so as to determine an attack text image pair of each field; inputting the attack text image pair into the to-be-tested large model for each field to obtain reply information of each field; and analyzing the reply information of each field by adopting a pre-constructed intelligent analysis model, and determining a comprehensive security assessment report of the to-be-tested large model in the plurality of fields. According to the invention, the accuracy of the security assessment result of the large model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of large model safety assessment, for example, to a safety assessment method, a safety assessment device and an electronic device for large models. Background Art

[0002] At present, large models have been widely used in many fields such as natural language processing, dialogue systems and generative tasks. However, if large models lack effective protection capabilities when facing potential security risks, their output results may contain sensitive information or harmful content when encountering improper input or malicious attacks. Therefore, it is necessary to conduct security assessments on large models.

[0003] In related technologies, the security indicators of large models are usually evaluated through template attacks or translation attacks.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] In the related art, test texts are used to conduct security assessments on large models, ignoring the fact that attackers may use image input to attack large models. When evaluating large models in different fields, it is ignored that attackers may use the characteristics of roles in different fields to attack large models by cleverly rewriting test data (for example, giving the large model the role of an accountant and asking the large model for sensitive tax information). This results in low accuracy of the security assessment results for large models. Therefore, how to improve the accuracy of the security assessment results for large models has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0007] The embodiments of the present disclosure provide a security assessment method, a security assessment device, and an electronic device for a large model, which can improve the accuracy of the security assessment results of the large model.

[0008] In some embodiments, a security assessment method for a large model includes: obtaining test text-image pairs for multiple fields; wherein the test text-image pairs include test text and test images; determining attack text based on the description information of the role corresponding to each field and the test text, and determining the attack image based on the test image and its corresponding attack text, so as to determine the attack text-image pair for each field; for each field, inputting the attack text-image pair into the large model to be tested, and obtaining response information for each field; using a pre-built intelligent analysis model to analyze the response information for each field, and determining a comprehensive security assessment report for the large model to be tested in multiple fields.

[0009] Optionally, obtaining test text-image pairs for multiple fields includes: receiving test data input by a user; determining the field to which each paragraph of text belongs based on the semantic information of each paragraph of text in the test data, and determining the field to which each image belongs based on the image content of each image in the test data; and associating and storing text and images belonging to the same field to determine test text-image pairs for multiple fields.

[0010] Optionally, for each field, the attack text image pair is input into the large model to be tested, and the reply information of each field is obtained, including: for each field, after the attack text image pair is input into the large model to be tested, identifying the situation that the current round reply information of the large model to be tested includes rejection words; when the current round reply information includes rejection words, modify the attack text image pair according to the set strategy and then input the attack text image pair into the large model to be tested again; when the current round reply information does not include rejection words, or when the number of rounds of inputting the attack text image pair into the large model to be tested reaches the set number of rounds, stopping the iteration, and using the reply information of all rounds and the modified attack text image pair as the reply information.

[0011] Optionally, before determining the attack text according to the description information of the role corresponding to each field and the test text, the security assessment method further includes: preprocessing the test text-image pair of each field.

[0012] Optionally, the test text image pair for each field is preprocessed, including: cleaning the test text in the test text image pair, and smoothing the test image in the test text image pair; verifying the text-graphic consistency of the test text and the test image in the processed test text image pair; and generating a new test image based on the test text when the test text and the test image in the test text image pair are inconsistent.

[0013] Optionally, a new test image is generated based on the test text, including: initializing a Gaussian noise image and converting the test text into a text feature vector; at each time step, determining the predicted noise corresponding to the current time step based on the text feature vector and the noise level corresponding to the current time step; and gradually removing the predicted noise from the Gaussian noise image according to the time step to obtain a new test image.

[0014] Optionally, after determining the comprehensive safety assessment report of the large model to be tested in multiple fields, the safety assessment method further includes: using a set display strategy to display the comprehensive safety assessment report.

[0015] Optionally, after determining a comprehensive safety assessment report of the large model to be tested in multiple fields, the safety assessment method further includes: optimizing the large model to be tested according to the comprehensive safety assessment report.

[0016] In some embodiments, a security assessment device for a large model includes a processor and a memory storing program instructions, and the processor is configured to execute the security assessment method for a large model as described above.

[0017] In some embodiments, an electronic device includes: a device body; and a safety assessment device for a large model as described above, which is arranged on the device body.

[0018] The safety assessment method, safety assessment device and electronic device for a large model provided by the embodiments of the present disclosure can achieve the following technical effects:

[0019] In the disclosed embodiment, when performing a security assessment on a large model, the test data used is a test text-image pair consisting of a test text and a test image, and the test text will be rewritten according to the description information of the corresponding role in each field to determine the attack text, and the attack image will be rewritten according to the attack text to determine the attack image. Then, the attack text-image pair consisting of the attack text and the attack image will be input into the large model to be tested, and the reply information of the security assessment of the large model to be tested in each field will be obtained. Finally, the reply information is analyzed by a pre-built intelligent analysis model to determine a comprehensive security assessment report of the large model to be tested in multiple fields. In this way, the security performance of the large model to be tested can be evaluated when the attacker cleverly rewrites the test data in the form of image input and by utilizing the characteristics of roles in different fields, thereby realizing a security assessment of the large model from a multi-modal perspective. Therefore, the disclosed embodiment can improve the accuracy of the security assessment results of the large model.

[0020] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0022] Figure 1 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a security assessment method for a large model provided by an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of another security assessment method for a large model provided by an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of another security assessment method for a large model provided by an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a safety assessment device for a large model provided by an embodiment of the present disclosure;

[0027] Figure 6 It is a schematic diagram of another safety assessment device for a large model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0029] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0030] Unless otherwise stated, the term "plurality" of a feature means two or more.

[0031] In the embodiment of the present disclosure, the character " / " feature indicates that the preceding and following objects are in an "or" relationship. For example, the A / B feature indicates: A or B.

[0032] The term "and / or" is a description of the association relationship between objects, and the characteristic indicates that there can be three relationships. For example, A and / or B, the characteristic indicates: A or B, or, A and B.

[0033] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0035] like Figure 1 As shown, the electronic device 100 provided by the embodiment of the present disclosure includes a device body 110 and a safety assessment device 500 (600) for a large model.

[0036] Specifically, the safety assessment device 500 ( 600 ) for a large model is disposed on the device body 110 .

[0037] Optionally, the security assessment device 600 for the large model includes a processor, which can obtain test text image pairs (test text image pairs include test text and test image) for security assessment of the large model to be tested in multiple fields. The attack text can be determined based on the description information of the role corresponding to each field and the test text, and the attack image can be determined based on the test image and its corresponding attack text to determine the attack text image pair for each field. For each field, the processor can input the attack text image pair into the large model to be tested, obtain the response information of each field, and use the pre-built intelligent analysis model to analyze the response information of each field to determine the comprehensive security assessment report of the large model to be tested in multiple fields.

[0038] In combination with the above electronic device, the present disclosure provides a security assessment method for a large model, such as Figure 2 As shown, the safety assessment method includes:

[0039] S201, the processor obtains test text-image pairs in multiple fields; wherein the test text-image pairs include test texts and test images.

[0040] It is understandable that large models usually involve applications in multiple fields, such as finance, education, and health. In these fields, sensitive information or harmful content may be output due to improper input or malicious attacks. Therefore, when conducting a security assessment on the large model to be tested, it is necessary to conduct testing and evaluation in these fields. Therefore, the processor needs to obtain test text image pairs in each field.

[0041] Specifically, the multiple fields include but are not limited to: finance, violence, pornography, politics, education, science and technology, health, entertainment, sports, tourism, culture and society, etc.

[0042] Specifically, an attacker may attack the large model by using image input, causing the large model to output sensitive information or harmful content. Therefore, for each field, the processor obtains a test text-image pair including both a test text and a test image.

[0043] S202, the processor determines the attack text according to the description information of the role corresponding to each field and the test text, and determines the attack image according to the test image and its corresponding attack text, so as to determine the attack text-image pair for each field.

[0044] Specifically, the description information of the role corresponding to each field refers to the role description information in this field that will cause the big model to output sensitive information or harmful content after assigning the role to the big model. For example, the description information of the accountant role in the financial field, the description information of the lawyer in the legal field, etc.

[0045] Specifically, since attackers may use the characteristics of roles in different fields to cleverly rewrite test data to attack the large model, causing the large model to output sensitive information or harmful content, after obtaining test text-image pairs in multiple fields, the processor adopts a role-playing attack strategy, determines the attack text based on the description information of the role corresponding to each field and the test text, and determines the attack image based on the test image and its corresponding attack text, so as to determine the attack text-image pair for each field.

[0046] Optionally, a text generation model is pre-built, and the text generation model generates attack text according to the description information of the role corresponding to each field and the test text. The generation process is shown in the following expression:

[0047] T = G(R,Q);

[0048] Among them, T is the attack text, G is the text generation model, R is the description information of the role corresponding to each field, and Q is the test text.

[0049] Optionally, after the attack text is generated, the similarity between the attack text and the test text needs to be checked. After the similarity score is greater than the set score, the attack text is determined to be usable. Otherwise, the above step of generating the attack text is performed again.

[0050] Optionally, the attack text is implicitly embedded into the test image using an image steganography strategy to obtain an attack image. The generation process is shown in the following expression:

[0051] I′=E(I,T);

[0052] Among them, I′ is the attack image, E is the embedding process, I is the test image, and T is the attack text.

[0053] Optionally, to ensure that the embedded attack text is visually invisible, an evaluation function is used to measure the visual difference between the attack image and the test image.

[0054] Specifically, the visual difference between the attack image and the test image is measured according to the following expression:

[0055] T(I,I′)=Visibility(I,I′);

[0056] Where V(I,I′) is the visual difference between the attack image and the test image, and Visibility is the metric function. If the visual difference is less than the set threshold, the embedded attack text is considered to be visually invisible. In this case, the attack image is confirmed to be usable. Otherwise, the above steps of generating the attack image are repeated.

[0057] S203, the processor inputs the attack text image pair into the large model to be tested for each field, and obtains the response information of each field.

[0058] Specifically, the attack text-image pair is input into the large model to be tested, and the process of obtaining the reply information is as shown in the expression:

[0059] Ans=M T (T,I′);

[0060] Among them, Ans is the reply information, M T is the large model to be tested, T is the attack text, and I′ is the attack image.

[0061] Optionally, for each field, the attack text image pair is input into the large model to be tested, and the reply information of each field is obtained, including: for each field, after the attack text image pair is input into the large model to be tested, identifying the situation that the current round reply information of the large model to be tested includes rejection words; when the current round reply information includes rejection words, modify the attack text image pair according to the set strategy and then input the attack text image pair into the large model to be tested again; when the current round reply information does not include rejection words, or when the number of rounds of inputting the attack text image pair into the large model to be tested reaches the set number of rounds, stopping the iteration, and using the reply information of all rounds and the modified attack text image pair as the reply information.

[0062] Specifically, the rejection words are used to indicate that the large model to be tested refuses to output sensitive information or harmful content in the reply information.

[0063] Specifically, a rejection vocabulary list is pre-built, and whether each round of reply information includes a rejection vocabulary list can be determined by identifying whether each round of reply information includes a word in the rejection vocabulary list.

[0064] Illustratively, a portion of the rejected vocabulary looks like this:

[0065] [“is illegal and unethical”, “I’m really sorry”, “immoral”, “illegal”, “illegal”, “I’m sorry”, “but I can’t”, “if you have any other non-malicious requests”, “I can’t provide”, “I’m really sorry”, “I can’t create”, “I can’t support”, “I can’t provide”].

[0066] Specifically, for each field, the attack text image pair is input into the large model to be tested for processing. Conducting security assessment of the large model to be tested in this field is an iterative process.

[0067] Specifically, after the attack text image pair of this round is input into the large model to be tested, if the output reply information of this round includes rejection words, it means that the large model to be tested can cope with the attack of the attack text image pair in this way. Therefore, in this case, it is necessary to modify the attack text image pair according to the set strategy and input the attack text image pair into the large model to be tested again to continue the test.

[0068] Specifically, after the attack text image pair of this round is input into the large model to be tested, if the output reply information of this round does not include rejection words, it means that the large model to be tested cannot cope with the attack of the attack text image pair of this round, and it can be determined that the large model to be tested has defects in coping with such attacks. Therefore, in this case, the iteration can be stopped directly.

[0069] Specifically, if the number of rounds of inputting attack text-image pairs into the large model to be tested reaches the set number of rounds, it indicates that the large model to be tested can cope with such attacks and has no defects in coping with such attacks. Therefore, in this case, the iteration can be stopped.

[0070] Specifically, after the iteration is stopped, the reply information of all rounds and the modified attack text-image pair are used as reply information. Based on the reply information, it can be determined whether the large model to be tested can successfully resist the attack in each field, as well as the number of rounds of resistance to the attack and other information.

[0071] S204, the processor uses a pre-built intelligent analysis model to analyze the response information of each field, and determines a comprehensive security assessment report of the large model to be tested in multiple fields.

[0072] Specifically, based on the response information output by the large model under test after inputting the attack text image pairs in each field, it can be determined whether the large model under test outputs sensitive information or harmful content when responding to attacks in these fields. Therefore, by using the pre-built intelligent analysis model to analyze the response information in each field, a comprehensive security assessment report of the large model under test in multiple fields can be determined.

[0073] In the disclosed embodiment, when performing a security assessment on a large model, the test data used is a test text-image pair consisting of a test text and a test image, and the test text will be rewritten according to the description information of the corresponding role in each field to determine the attack text, and the attack image will be rewritten according to the attack text to determine the attack image. Then, the attack text-image pair consisting of the attack text and the attack image will be input into the large model to be tested, and the reply information of the security assessment of the large model to be tested in each field will be obtained. Finally, the reply information is analyzed by a pre-built intelligent analysis model to determine a comprehensive security assessment report of the large model to be tested in multiple fields. In this way, the security performance of the large model to be tested can be evaluated when the attacker cleverly rewrites the test data in the form of image input and by utilizing the characteristics of roles in different fields, thereby realizing a security assessment of the large model from a multi-modal perspective. Therefore, the disclosed embodiment can improve the accuracy of the security assessment results of the large model.

[0074] In some embodiments, obtaining test text-image pairs for multiple fields includes: receiving test data input by a user; determining the field to which each paragraph of text belongs based on the semantic information of each paragraph of text in the test data, and determining the field to which each image belongs based on the image content of each image in the test data; and associating and storing text and images belonging to the same field to determine test text-image pairs for multiple fields.

[0075] Specifically, since the fields involved in the large model to be tested may be only a few specific fields, after receiving the test data input by the user, the test data needs to be divided according to the fields.

[0076] Specifically, the semantic information of each text in the test data can be used to determine which field the text expresses. Therefore, the field to which each text belongs can be determined based on the semantic information of each text.

[0077] For example, a text is: As the Federal Reserve announced an interest rate hike, global stock markets fell, and investors expressed concerns about the future economic trend. It can be determined that this text belongs to the test text in the financial field based on its semantic information.

[0078] For example, a text is: The latest research shows that regular exercise can significantly reduce the risk of heart disease, and experts recommend that adults do at least 150 minutes of moderate-intensity exercise per week. It can be determined that this text belongs to the test text in the health field based on its semantic information.

[0079] For example, a text is: In last night's game, football player Xiao Ming performed a hat trick, helping the team beat the opponent 3-1 and winning warm applause from the fans. It can be determined that the text belongs to the test text in the sports field based on its semantic information.

[0080] Specifically, according to the image content of each image in the test data, it can be determined which field the image expresses. Therefore, according to the image content of each image, it is possible to determine the field to which each image belongs.

[0081] For example, taking an image showing currencies of multiple countries (including US dollars, euros and yen) and financial charts as an example, it can be determined that the image belongs to a test image in the financial field based on its image content.

[0082] For example, take an image depicting a fierce confrontation in a boxing match where two players are punching with all their strength as an example, then it can be determined based on its image content that the image belongs to a test image in the field of sports.

[0083] For example, take an image showing ancient artworks, such as sculptures and paintings, in a museum exhibition as an example, then it can be determined that the image belongs to the test image in the cultural field based on its image content.

[0084] Optionally, a pre-built multimodal graphic and text model can be used to analyze the test data, classify the test text and test image data in the test data, and store them in an associated manner, so as to process the test data input by the user into test text-image pairs in multiple fields.

[0085] Optionally, the test data can be processed in advance, and then the test text image pairs obtained in multiple fields are stored in a preset database classification structure. Each test text image pair is assigned a unique identifier and stored in a corresponding classification table according to the classification results of its field (such as finance, violence, pornography, etc.). For example, test text image pairs related to the financial field are stored in the financial classification table. In this way, when performing a security assessment on the large model to be tested, the processor can directly call the test text image pairs of the corresponding field from the database, reducing the process of preliminary test data processing, which is conducive to improving the efficiency of security assessment of the large model to be tested.

[0086] In addition, after storing the test text-image pairs in multiple fields in a preset database classification structure, data backup can be performed regularly, and security measures such as encrypted storage and access control can be implemented to ensure the security and integrity of the test text-image pairs.

[0087] The disclosed embodiment provides another security assessment method for large models, such as Figure 3 As shown, the safety assessment method includes:

[0088] S301, the processor obtains test text-image pairs in multiple fields; wherein the test text-image pairs include test texts and test images.

[0089] S302: The processor pre-processes the test text image pairs in each field.

[0090] It is understandable that the test text and the test image may include some noise data, which may affect the subsequent conversion of the attack text and the attack image. Therefore, the test text needs to be preprocessed first.

[0091] Optionally, the test text image pair for each field is preprocessed, including: cleaning the test text in the test text image pair, and smoothing the test image in the test text image pair; verifying the text-graphic consistency of the test text and the test image in the processed test text image pair; and generating a new test image based on the test text when the test text and the test image in the test text image pair are inconsistent.

[0092] Specifically, the purpose of cleaning the test text is to remove irrelevant information in the test text and improve the data quality of the test text. The steps of text cleaning include: removing blank characters at the beginning and end of the test text, replacing consecutive blank characters with a space; deleting non-printing characters such as tabs, line feeds, and carriage returns; removing or replacing special symbols as needed, removing numbers in the text, etc.

[0093] Specifically, the purpose of smoothing the test image is to improve the image quality of the test image. The smoothing steps include: scaling and cropping the test image to adjust the size of the test image; and using a Gaussian filter to perform weighted average filtering on the resized test image.

[0094] Specifically, during the weighted average filtering process, the weighted average value between each pixel of the test image and its neighboring pixels is calculated according to the following expression to obtain the denoised pixel value.

[0095]

[0096] Among them, I′(x,y) is the pixel value of the test image at point (x,y) after filtering, I(x+i,y+j) is the pixel value of the test image at point (x+i,y+j) before filtering, and G(i,j) is the weight of the Gaussian filter kernel at position (i,j).

[0097] Specifically, if the test text and the test image in the test text image pair are inconsistent, it will affect the subsequent security assessment. Therefore, after cleaning the test text and smoothing the test image, it is necessary to verify the consistency of the test text and the test image. And in the case of consistency, a new test image needs to be generated based on the test text.

[0098] Optionally, the image-text consistency of the test text and the test image is verified by calculating the similarity between the image features of the test image and the text features of the test text.

[0099] Specifically, the CLIP (Contrastive Language-Image Pre-training) model is used to calculate the similarity between the image features of the test image and the text features of the test text through the following expression.

[0100]

[0101] Among them, Similarity is the similarity, v I is the image feature of the test image, v T is the text feature of the test text.

[0102] Specifically, if the similarity is greater than or equal to a set threshold, it is determined that the text and graphics of the test text and the test image are consistent.

[0103] Optionally, a new test image is generated based on the test text, including: initializing a Gaussian noise image and converting the test text into a text feature vector; at each time step, determining the predicted noise corresponding to the current time step based on the text feature vector and the noise level corresponding to the current time step; and gradually removing the predicted noise from the Gaussian noise image according to the time step to obtain a new test image.

[0104] Specifically, a Diffusion model may be used to generate a new test image according to the test text.

[0105] Specifically, the process of gradually removing the prediction noise from the Gaussian noise image according to the time step to obtain a new test image is expressed by the following expression.

[0106]

[0107] Among them, z θ (xt ,t,v T ) is the prediction noise, x t is a Gaussian noise image, v T is the text feature vector, x t-1 Gaussian noise image after processing according to the predicted noise corresponding to time step t.

[0108] Specifically, the Gaussian noise image obtained at time step t=0 is the new test image.

[0109] S303, the processor determines the attack text according to the description information of the role corresponding to each field and the test text, and determines the attack image according to the test image and its corresponding attack text, so as to determine the attack text-image pair for each field.

[0110] S304, the processor inputs the attack text-image pair into the large model to be tested for each field, and obtains response information for each field.

[0111] S305, the processor uses a pre-built intelligent analysis model to analyze the response information of each field, and determines a comprehensive security assessment report of the large model to be tested in multiple fields.

[0112] In the disclosed embodiment, after obtaining test text image pairs in multiple fields, the test text image pairs in each field are preprocessed and then converted into attack text image pairs, which is helpful to improve the usability of the determined attack text image pairs.

[0113] In some embodiments, after determining the comprehensive safety assessment report of the large model to be tested in multiple fields, the safety assessment method further includes: using a set display strategy to display the comprehensive safety assessment report.

[0114] Specifically, the display strategy is to classify the comprehensive security assessment report into different fields (such as financial fraud, violent behavior, and pornography), and list the attack text-image pairs, attack times, and attack success rates in each field in detail. This makes it easier for users to fully understand the security status of the large model to be tested against attacks in various fields.

[0115] Specifically, in order to present these classified data more intuitively, the comprehensive security assessment report can be presented in the form of performance indicator charts (such as line charts, bar charts, and pie charts). In this way, the distribution and success rate of attack text-image pairs in various fields can be clearly and intuitively displayed. Through this display method, it is easy for users to intuitively understand the proportion of attacks in different fields in the overall, and the potential defects and deficiencies of the large model to be tested in different fields.

[0116] Specifically, the changes in the data of the large model to be tested before and after being attacked can be compared and displayed. For example, the response of the large model to be tested to the tested text image pair and the attacking text image pair can be displayed. The following presentation methods can be used:

[0117] Text comparison: For the responses to the test text and attack text, highlight the differences or add sidebar comments.

[0118] Image comparison: For the reply information of the test image and the attack image, the image overlay or split-screen display method is used to intuitively display the difference in image processing results.

[0119] The disclosed embodiment provides another security assessment method for large models, such as Figure 4 As shown, the safety assessment method includes:

[0120] S401, the processor obtains test text-image pairs in multiple fields; wherein the test text-image pairs include test texts and test images.

[0121] S402, the processor determines the attack text according to the description information of the role corresponding to each field and the test text, and determines the attack image according to the test image and its corresponding attack text, so as to determine the attack text-image pair for each field.

[0122] S403, the processor inputs the attack text image pair into the large model to be tested for each field, and obtains the response information of each field.

[0123] S404, the processor uses a pre-built intelligent analysis model to analyze the response information of each field, and determines a comprehensive security assessment report of the large model to be tested in multiple fields.

[0124] S405: The processor optimizes the large model to be tested according to the comprehensive safety assessment report.

[0125] Specifically, the insufficiency of the large model to be tested in the face of test data in various fields can be determined according to the comprehensive safety assessment report. Therefore, the processor can optimize the large model to be tested according to the comprehensive safety assessment report.

[0126] In the disclosed embodiment, after the comprehensive security assessment report of the large model to be tested in multiple fields is determined, the large model to be tested is optimized according to the comprehensive security assessment report, thereby improving the security of the application of the large model to be tested.

[0127] Combination Figure 5As shown, the embodiment of the present disclosure provides a security assessment device 500 for a large model, including: an acquisition module 501, a processing module 502 and an analysis module 503. The acquisition module 501 is configured to acquire test text image pairs in multiple fields; wherein the test text image pairs include test texts and test images. The processing module 502 is configured to determine the attack text according to the description information of the role corresponding to each field and the test text, and determine the attack image according to the test image and its corresponding attack text, so as to determine the attack text image pair for each field. The processing module 502 is also configured to input the attack text image pair into the large model to be tested for each field, and obtain the response information of each field. The analysis module 503 is configured to use a pre-built intelligent analysis model to analyze the response information of each field, and determine a comprehensive security assessment report of the large model to be tested in multiple fields.

[0128] Combination Figure 6 As shown, the embodiment of the present disclosure provides a security assessment device 600 for a large model, including: a processor (processor) 601 and a memory (memory) 602. Optionally, the device may also include a communication interface (Communication Interface) 603 and a bus 604. Among them, the processor 601, the communication interface 603, and the memory 602 can communicate with each other through the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call the logic instructions in the memory 602 to execute the security assessment method for the large model of the above embodiment.

[0129] In addition, the logic instructions in the memory 602 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0130] The memory 602 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 601 executes the function application and data processing by running the program instructions / modules stored in the memory 602, that is, the security assessment method for large models in the above embodiment is implemented.

[0131] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory.

[0132] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned security assessment method for a large model.

[0133] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0134] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0136] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0137] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A safety assessment method for a large model, characterized in that: include: Acquire test text-image pairs in multiple fields; wherein the test text-image pairs include test text and test images; Determine the attack text according to the description information of the role corresponding to each field and the test text, and determine the attack image according to the test image and its corresponding attack text, so as to determine the attack text-image pair for each field; For each domain, input the attack text-image pair into the large model to be tested, and obtain the response information of each domain; A pre-built intelligent analysis model is used to analyze the response information in each field to determine a comprehensive security assessment report for the large model to be tested in multiple fields.

2. The security assessment method according to claim 1, characterized in that: Get test text-image pairs from multiple domains, including: Receive test data input by the user; Determine the domain to which each text belongs according to the semantic information of each text in the test data, and determine the domain to which each image belongs according to the image content of each image in the test data; The texts and images belonging to the same field are stored in association, and test text-image pairs in multiple fields are determined.

3. The safety assessment method according to claim 1, characterized in that: For each domain, the attack text-image pair is input into the large model to be tested, and the response information of each domain is obtained, including: For each domain, after inputting the attack text-image pair into the large model to be tested, identify the situation where the current round of reply information of the large model to be tested includes rejected words; When the reply information in this round includes rejected words, the attack text-image pair is modified according to the set strategy and then input into the large model to be tested again; When the reply information of this round does not include rejected words, or when the number of rounds of inputting attack text-image pairs into the large model to be tested reaches the set number of rounds, the iteration is stopped and the reply information of all rounds and the modified attack text-image pairs are used as reply information.

4. The safety assessment method according to any one of claims 1 to 3, characterized in that: Before determining the attack text based on the description information and test text of the corresponding role in each field, the security assessment method also includes: Preprocess the test text-image pairs for each domain.

5. The safety assessment method according to claim 4, characterized in that: Preprocess the test text-image pairs in each domain, including: Performing a cleaning process on the test text in the test text image pair, and performing a smoothing process on the test image in the test text image pair; Verify the text-image consistency of the test text and the test image in the processed test text-image pair; When the test text and the test image in the test text-image pair are inconsistent, a new test image is generated according to the test text.

6. The safety assessment method according to claim 5, characterized in that: Generate new test images based on test text, including: Initialize the Gaussian noise image and convert the test text into a text feature vector; At each time step, the predicted noise corresponding to the current time step is determined based on the text feature vector and the noise level corresponding to the current time step; According to the time step, the prediction noise is gradually removed from the Gaussian noise image to obtain a new test image.

7. The safety assessment method according to any one of claims 1 to 3, characterized in that: After determining the comprehensive safety assessment report of the large model to be tested in multiple areas, the safety assessment method also includes: The comprehensive safety assessment report is presented using the set presentation strategy.

8. The safety assessment method according to any one of claims 1 to 3, characterized in that: After determining the comprehensive safety assessment report of the large model to be tested in multiple areas, the safety assessment method also includes: Based on the comprehensive safety assessment report, the large model under test is optimized.

9. A safety assessment device for a large model, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to be able to execute the safety assessment method for a large model according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: Equipment body; The safety assessment device for a large model as described in claim 9 is arranged on the device body.

Images