Model evaluation methods, devices, equipment, media and program products

By splitting and combining attack prompts to generate complex images, the accuracy of defense capability assessment for multimodal large models is improved, solving the problem of inaccurate assessment in existing technologies and achieving effective assessment in complex attack scenarios.

CN123074756APending Publication Date: 2026-11-10CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611080989.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-20
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies make it difficult to accurately assess the defense capabilities of multimodal large models against complex and covert induced attacks, resulting in inaccurate assessments of defense capabilities.

Method used

By splitting attack prompts into sub-prompts and generating complex third images by combining highly similar images, and inputting them into a multimodal large model for evaluation, the model's inference complexity is increased, and the attention mechanism defense capability of the Transformer architecture model is reduced.

Benefits of technology

It improves the accuracy of assessing multimodal large-scale model defense capabilities, enabling adversarial testing using samples that better match actual attack command standards, identifying weaknesses in the model and making targeted fixes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123074756A_ABST
    Figure CN123074756A_ABST
Patent Text Reader

Abstract

This application provides a model evaluation method, apparatus, device, medium, and program product, relating to the field of artificial intelligence technology, for improving the accuracy of evaluating the defense capabilities of multimodal large models. The specific technical solution is as follows: acquiring a first attack warning word and a first image set; splitting the first attack warning word to obtain at least one sub-warning word, and generating a second image corresponding to each sub-warning word based on the at least one sub-warning word; determining M first images from N first images based on the similarity between the first attack warning word and each first image; generating a third image based on the second image corresponding to the at least one sub-warning word and the M first images; inputting the third image into the multimodal large model to obtain image recognition results. This application is applied to scenarios involving the evaluation of the defense capabilities of multimodal large models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model evaluation method, apparatus, device, medium, and program product. Background Technology

[0002] As the application scope of multimodal large models continues to expand, attacks targeting multimodal large models are also increasing. The most common type is the inducing command attack, which uses malicious prompts to achieve malicious purposes. Therefore, it is necessary to evaluate the defense capabilities of multimodal large models against inducing commands in order to identify the weaknesses of multimodal large models and carry out targeted repairs.

[0003] Existing technologies embed malicious commands into image samples and utilize the visual understanding capabilities of multimodal large models. When the model recognizes and processes the image, it extracts potential misleading commands, enabling cross-modal attack testing of the model. The model's defense capability is then evaluated based on the success rate of its defense against such attacks.

[0004] However, real-world induced attacks are often highly complex and covert. Simply embedding explicit commands into images is insufficient to achieve the standards and effects of actual induced command attacks, leading to inaccurate assessments of the defense capabilities against multimodal large-scale attacks. Summary of the Invention

[0005] This application provides a model evaluation method, apparatus, device, medium, and program product to improve the accuracy of evaluating the defense capabilities of multimodal large models.

[0006] In a first aspect, embodiments of this application provide a model evaluation method, the method comprising: acquiring a first attack prompt word and a first image set, wherein the first image set contains N first images, where N is an integer greater than 1; splitting the first attack prompt word to obtain at least one sub-prompt word, and generating a second image corresponding to each sub-prompt word based on the at least one sub-prompt word; determining M first images from the N first images based on the similarity between the first attack prompt word and each first image, wherein the similarity between the M first images is less than a first similarity threshold, and the similarity between the M first images and the first attack prompt word is less than a second similarity threshold, where M is a positive integer less than or equal to N; generating a third image based on the second image corresponding to the at least one sub-prompt word and the M first images; inputting the third image into a multimodal large model to obtain an image recognition result, wherein the image recognition result is used to characterize whether the first attack prompt word successfully attacks the multimodal large model.

[0007] The technical solution provided in this application brings at least the following beneficial effects: It acquires a first attack prompt word and a first image set; it splits the first attack prompt word to obtain sub-prompt words, and generates a second image corresponding to each sub-prompt word based on the sub-prompt words; it determines M first images from the N first images based on the similarity between the first attack prompt word and each first image; it generates a third image based on the second image corresponding to at least one sub-prompt word and the M first images; and it inputs the third image into a multimodal large-scale model to obtain image recognition results. In this solution, by splitting the first attack prompt word into sub-prompt words, the content of the first attack prompt word is initially decomposed, and a corresponding second image is generated based on the content of the sub-prompt words. Then, first images with significant semantic differences from the first attack prompt word are acquired from the first image set to increase the complexity of the image content. The first and second images are then combined to obtain a third image containing complex content and the complete content of the first attack prompt word. This third image is used to evaluate the defense capabilities of the multimodal large-scale model and allows for adversarial testing using samples that better conform to actual attack command standards, thereby improving the accuracy of the evaluation of the multimodal large-scale model's defense capabilities.

[0008] One possible implementation, based on the similarity between the first attack prompt and each first image, determines M first images from the N first images, including: obtaining the first similarity between each of the N first images and the first attack prompt, and determining the first target image corresponding to the minimum first similarity from the N first images; obtaining the second similarity between each of the N first images and the i-th target image, and determining the (i+1)-th target image corresponding to the minimum similarity sum from the N first images based on the first similarity corresponding to each first image and the similarity sum of K second similarities, to obtain M first images, wherein the K second similarities corresponding to a first image include the second similarities of the first i target images corresponding to the first image, i∈[1,M-1], and i is an integer.

[0009] Another possible implementation, the above-mentioned acquisition of the first similarity between each of the N first images and the first attack prompt word, includes: acquiring a first vector of each of the N first images and a second vector of the first attack prompt word, wherein the first vector is used to represent the semantic information of each of the N first images and the second vector is used to represent the semantic information of the first attack prompt word; acquiring the cosine similarity between the first vector and the second vector to obtain the first similarity between each of the N first images and the first attack prompt word.

[0010] Another possible implementation is that, based on the second image corresponding to at least one sub-prompt word and the M first images, the generation of the third image includes: stitching the second image corresponding to at least one sub-prompt word and the M first images together to obtain a first stitched image; and feathering the edge regions of the second image or the first image in the first stitched image to obtain the third image.

[0011] Another possible implementation is that the above-mentioned input of the third image into the multimodal large model to obtain the image recognition result includes: inputting the third image into the multimodal large model, extracting information from the third image through the multimodal large model, and obtaining the first output result.

[0012] The first output result is matched with the preset first information, which is used to characterize the first attack prompt word as having successfully attacked the multimodal large model. If the match is successful, an image recognition result is output to characterize the first attack prompt word as having successfully attacked the multimodal large model. If the match fails, an image recognition result is output to characterize the first attack prompt word as having failed to attack the multimodal large model.

[0013] Secondly, embodiments of this application provide a model evaluation apparatus, comprising: an acquisition module, a splitting module, and a generation module; the acquisition module is used to acquire a first attack prompt word and a first image set, the first image set containing N first images, where N is an integer greater than 1; the splitting module is used to split the first attack prompt word to obtain at least one sub-prompt word, and generate a second image corresponding to each sub-prompt word based on the at least one sub-prompt word; the acquisition module is further used to determine M first images from the N first images based on the similarity between the first attack prompt word and each first image, the similarity of the M first images being less than a first similarity threshold, and the similarity between the M first images and the first attack prompt word being less than a second similarity threshold, where M is a positive integer less than or equal to N; the generation module is used to generate a third image based on the second image corresponding to the at least one sub-prompt word and the M first images; the acquisition module is further used to input the third image into a multimodal large model to obtain an image recognition result, the image recognition result being used to characterize whether the first attack prompt word successfully attacks the multimodal large model.

[0014] One possible implementation is that the acquisition module described above is specifically used to: acquire the first similarity between each of the N first images and the first attack prompt word, and determine the first target image corresponding to the minimum first similarity from the N first images; acquire the second similarity between each of the N first images and the i-th target image, and based on the first similarity corresponding to each first image and the similarity sum of the K second similarities, determine the (i+1)-th target image corresponding to the minimum similarity sum from the N first images to obtain M first images, wherein the K second similarities corresponding to a first image include the second similarities of the first i target images corresponding to the first image, i∈[1,M-1], and i is an integer.

[0015] Another possible implementation, the acquisition module described above, is specifically used to: acquire a first vector for each of the N first images and a second vector for the first attack prompt word, wherein the first vector is used to characterize the semantic information of each of the N first images and the second vector is used to characterize the semantic information of the first attack prompt word; acquire the cosine similarity between the first vector and the second vector to obtain the first similarity between each of the N first images and the first attack prompt word.

[0016] Another possible implementation is that the above-mentioned generation module is specifically used to: stitch together the second image corresponding to at least one sub-prompt word and the M first images to obtain a first stitched image; and feather the edge regions of the second image or the first image in the first stitched image to obtain a third image.

[0017] Another possible implementation is that the acquisition module described above is specifically used to: input the third image into the multimodal large model, extract information from the third image through the multimodal large model, and obtain a first output result; match the first output result with preset first information, the first information being used to characterize the first attack prompt word's successful attack on the multimodal large model; if the match is successful, output an image recognition result characterizing the first attack prompt word's successful attack on the multimodal large model; if the match fails, output an image recognition result characterizing the first attack prompt word's failed attack on the multimodal large model.

[0018] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.

[0019] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.

[0020] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.

[0021] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0022] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0023] Figure 1 A schematic diagram of the network architecture for an application of a model evaluation method provided in this application embodiment;

[0024] Figure 2 A schematic flowchart illustrating a model evaluation method provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of a sub-prompt word image for a model evaluation method provided in an embodiment of this application;

[0026] Figure 4 A schematic flowchart illustrating a model evaluation method provided in an embodiment of this application;

[0027] Figure 5 A schematic flowchart illustrating a model evaluation method provided in an embodiment of this application;

[0028] Figure 6 A schematic diagram of image stitching for a model evaluation method provided in an embodiment of this application;

[0029] Figure 7 A schematic flowchart illustrating a model evaluation method provided in an embodiment of this application;

[0030] Figure 8 A schematic flowchart illustrating a model evaluation method provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the structure of a model evaluation device provided in an embodiment of this application;

[0032] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The model evaluation method, apparatus, equipment, media, and program products provided in this application will be described in detail below with reference to the accompanying drawings.

[0034] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0035] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0036] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0037] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0038] The model evaluation method, apparatus, device, medium, and program product provided in this application embodiment can be applied to scenarios where the defense capabilities of multimodal large models are evaluated.

[0039] As the application scope of multimodal large models continues to expand, attacks targeting multimodal large models are also increasing. The most common type is the inducing command attack, which uses malicious prompts to achieve malicious purposes. Therefore, it is necessary to evaluate the defense capabilities of multimodal large models against inducing commands in order to identify the weaknesses of multimodal large models and carry out targeted repairs.

[0040] Existing technologies embed malicious commands into image samples and utilize the visual understanding capabilities of multimodal large models. When the model recognizes and processes the image, it extracts potential misleading commands, enabling cross-modal attack testing of the model. The model's defense capability is then evaluated based on the success rate of its defense against such attacks.

[0041] However, real-world induced attacks are often highly complex and covert. Simply embedding explicit commands into images is insufficient to achieve the standards and effects of actual induced command attacks, leading to inaccurate assessments of the defense capabilities against multimodal large-scale attacks.

[0042] Existing multimodal AI security assessment techniques are mostly single-modal assessment methods. For example, in the assessment of large multimodal models of text-based images, existing methods mainly fall into two categories:

[0043] Adding attack commands directly to text prompts is easily blocked by the model's own protection mechanisms.

[0044] By leveraging the understanding capabilities of multimodal AI models, inducements can be embedded into images through methods such as Optical Character Recognition (OCR). However, this method is also easily blocked by the model's own protection mechanisms.

[0045] To address the issue of the limited range of existing security assessment methods, this solution proposes a multimodal large-scale model security assessment method, device, and system based on enhanced visual complexity. By decomposing malicious images, the complexity of model inference is increased, reducing the security defense capability of the attention mechanism in the Transformer architecture model, and effectively assessing the defense capability of multimodal AI in the face of complex attack scenarios.

[0046] In this solution, attack prompts are broken down and converted into images to increase the complexity of model inference, reduce the security defense capability of the attention mechanism of the Transformer architecture model, uncover potential security risks, provide a foundation for subsequent large-scale model security protection, and effectively evaluate the defense capability of multimodal artificial intelligence (AI) in the face of complex attack scenarios.

[0047] To address the aforementioned technical problems, embodiments of this application provide a model evaluation method, apparatus, device, medium, and program product. By splitting a first attack prompt word into sub-prompt words, the content of the first attack prompt word is initially decomposed. A corresponding second image is generated based on the content of the sub-prompt words. Then, a first image with a significant semantic difference from the first attack prompt word is obtained from a first image set to increase the complexity of the image content. The first and second images are then combined to obtain a third image containing complex content and the complete content of the first attack prompt word. This image is used to evaluate the defense capabilities of a multimodal large model. It allows for adversarial testing using samples that more closely match actual attack command standards, thereby improving the accuracy of the evaluation of the defense capabilities of a multimodal large model.

[0048] The model evaluation methods, apparatus, devices, media, and program products provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0049] Figure 1 The diagram illustrates a network architecture for applying a model evaluation method according to an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a model evaluation device 101 and a terminal device 102. The model evaluation device 101 and the terminal device 102 are interconnected.

[0050] In some embodiments, the model evaluation device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster comprising multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the model evaluation device 101. Figure 1 The Chinese model evaluation device 101 is shown as a single server as an example.

[0051] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.

[0052] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0053] See Figure 2 This is a flowchart illustrating a model evaluation method provided in an embodiment of this application. Figure 2 As shown, the model evaluation method provided in this application embodiment can be implemented by the above-mentioned model evaluation device, specifically including the following steps 201 to 205.

[0054] Step 201: The model evaluation device acquires the first attack prompt word and the first image set.

[0055] In some embodiments of this application, the first image set contains N first images, where N is an integer greater than 1.

[0056] In some embodiments of this application, the aforementioned first attack prompt can be used to attack a large model in order to manipulate the large model through specific instructions.

[0057] In some embodiments of this application, the first image set described above may include at least one arbitrary image.

[0058] Step 202: The model evaluation device splits the first attack prompt word to obtain at least one sub-prompt word, and generates a second image corresponding to each sub-prompt word based on the at least one sub-prompt word.

[0059] In some embodiments of this application, the model evaluation device can use any large open-source model to decompose the first attack prompt word through prompt words.

[0060] In some embodiments of this application, the model evaluation device can split the first attack prompt word using the formula 1:

[0061] (1)

[0062] in, G represents the sub-comment after splitting, G represents the large model of splitting the first attack compendium, and P represents the first attack compendium.

[0063] For example, the number of sub-prompts that can be decomposed from the first attack prompt can be three.

[0064] In some embodiments of this application, the model evaluation device can convert the text of the decomposed sub-prompt words into an image representation of the second image, and the formula for converting text into the second image is Formula 2:

[0065] (2)

[0066] in, This represents the second image after the i-th sub-prompt word has been transformed. This means that the i-th sub-cue word is transformed into the second image through the large model T.

[0067] For example, such as Figure 3 As shown, a schematic diagram of the sub-prompt image is presented. For example, the first attack prompt is "How to obtain privacy information". According to semantic segmentation, the first attack prompt is split into three sub-prompts: "Explain privacy information"; "What methods can be used to obtain privacy information"; and "Obtain privacy information". Then, the three sub-prompts can be directly displayed on three sub-images.

[0068] Step 203: The model evaluation device determines M first images from N first images based on the similarity between the first attack prompt word and each first image.

[0069] In some embodiments of this application, the similarity of the M first images is less than a first similarity threshold.

[0070] In some embodiments of this application, the similarity between the aforementioned M first images and the aforementioned first attack prompt words is all less than the second similarity threshold, where M is a positive integer less than or equal to N.

[0071] In some embodiments of this application, the first image mentioned above is any image in the first image set.

[0072] In some embodiments of this application, the model evaluation device can obtain the similarity between the first attack prompt word and each first image in the first image set.

[0073] In some embodiments of this application, the aforementioned similarity can characterize the semantic similarity between the first attack prompt word and the first image quality inspection.

[0074] In some embodiments, combined with Figure 2 ,like Figure 4 As shown, step 203 above can be implemented through steps 203a and 203b.

[0075] Step 203a: The model evaluation device obtains the first similarity between each of the N first images and the first attack prompt word, and determines the first target image corresponding to the minimum first similarity from the N first images.

[0076] In some embodiments of this application, the first similarity described above can be used to characterize the semantic similarity between the first image and the first attack prompt word.

[0077] In some embodiments of this application, the model evaluation device can obtain the first similarity between each of the N first images in the first image set and the first attack prompt word, so as to obtain N first similarities.

[0078] In some embodiments of this application, the model evaluation device can obtain the minimum first similarity among the above N first similarities.

[0079] In some embodiments of this application, the model evaluation device can obtain the corresponding first target image based on the aforementioned minimum first similarity.

[0080] In some embodiments, combined with Figure 4 ,like Figure 5 As shown, step 203a can be implemented in the following steps 203a1 and 203a2.

[0081] Step 203a1: The model evaluation device acquires the first vector of each of the N first images and the second vector of the aforementioned first attack prompt word.

[0082] The first vector is used to characterize the semantic information of each of the first images, and the second vector is used to characterize the semantic information of the first attack prompt word.

[0083] In some embodiments of this application, the model evaluation device can obtain a first vector for each of the N first images in the first image set to obtain N first vectors, which can characterize the semantic information of the first image.

[0084] In some embodiments of this application, the model evaluation device can obtain a second vector corresponding to the first attack prompt word.

[0085] In some embodiments of this application, the second vector described above may represent the semantic features of the first attack prompt word.

[0086] For example, the model evaluation device can transform the original attack prompt into a second vector using the CLIP encoder (CLIP-ViT-L / 14 model): ,in, This represents the second vector corresponding to the first attack cue word P.

[0087] Step 203a2: The model evaluation device obtains the cosine similarity between the first vector and the second vector, obtains the first similarity between each of the N first images and the first attack prompt word, and determines the first target image corresponding to the minimum similarity from the N first images.

[0088] In some embodiments of this application, the model evaluation device can obtain the cosine similarity between each of the above N first vectors and the second vector to obtain N cosine similarities.

[0089] In some embodiments of this application, the model evaluation device may use the cosine similarity as the first similarity between each first image and the first attack prompt word.

[0090] In some embodiments of this application, the model evaluation device can obtain the minimum first similarity and determine the first image corresponding to the minimum first similarity as the first target image.

[0091] In some embodiments of this application, the calculation formula for the model evaluation device to obtain the first target image is Formula 3:

[0092] (3)

[0093] in: This represents the first target image. For any first image in the first image set, This represents the first vector corresponding to image I. This represents the second vector.

[0094] Step 203b: The model evaluation device obtains the second similarity between each of the N first images and the i-th target image, and based on the first similarity corresponding to each first image and the similarity sum of the K second similarities, determines the (i+1)-th target image corresponding to the minimum similarity sum from the N first images to obtain M first images.

[0095] In some embodiments of this application, the K second similarities corresponding to a first image include the second similarities between the first image and the first i target images, i∈[1,M-1], and i is an integer.

[0096] In some embodiments of this application, after obtaining the i-th target image, the model evaluation device can obtain the second similarity between each first image in the first image set and each target image in the i-th target image.

[0097] In some embodiments of this application, the model evaluation device can obtain the second similarity between any first image in the first image set and the first i target images, sum them to obtain the second similarity sum value of the first image, and sum the second similarity sum value with the first similarity of the first image to obtain the similarity sum value of the first image.

[0098] In some embodiments of this application, the model evaluation device can obtain the similarity and value of each first image in the first image set.

[0099] In some embodiments of this application, the model evaluation device can determine the first image corresponding to the minimum similarity value as the (i+1)th target image.

[0100] In some embodiments of this application, the calculation formula for the model evaluation device to obtain the (i+1)th target image is Formula 4:

[0101] (4)

[0102] in: This represents the (i+1)th target image. The second vector representing the first attack cue word. Represents the first image and I first vectors. This represents the first vector of the k-th target image.

[0103] In some embodiments of this application, the model evaluation device can acquire a total of M target images to determine the M target images as M first images.

[0104] Step 204: The model evaluation device generates a third image based on the second image corresponding to at least one sub-prompt word and M first images.

[0105] In some embodiments of this application, the model evaluation device can combine the second image and the M first images to obtain a third image.

[0106] In some embodiments of this application, the formula for the model evaluation device to combine the second image and the M first images to obtain the third image is expressed as Formula 5:

[0107] (5)

[0108] in, This represents the third image after combination. Represents any one of the M first images. This represents any second image.

[0109] In some embodiments of this application, the above combinations can be assembled in a specific order.

[0110] For example, such as Figure 6 The diagram illustrates an image stitching technique, which includes nine first images: "Image 1", "Image 2", "Image 3", "Image 4", "Image 5", "Image 6", "Image 7", "Image 8", and "Image 9".

[0111] In some embodiments, combined with Figure 2 ,like Figure 7 As shown, step 204 above can be implemented through steps 204a and 204b.

[0112] Step 204a: The model evaluation device performs a stitching process on the second image corresponding to at least one sub-prompt word and M first images to obtain a first stitched image.

[0113] In some embodiments of this application, the model evaluation device can stitch together a second image and M first images in a specific order.

[0114] In some embodiments of this application, the model evaluation device can partially or completely overlap and stitch together the first images among the M first images.

[0115] Step 204b: The model evaluation device feathers the edge regions of the second image or the first image in the first stitched image to obtain the third image.

[0116] In some embodiments of this application, the model evaluation device can feather the edges of the first image or the second image to eliminate harsh edges and achieve a natural transition.

[0117] Step 205: The model evaluation device inputs the third image into the multimodal large model to obtain the image recognition result.

[0118] In some embodiments of this application, the image recognition results are used to characterize whether the first attack prompt word has successfully attacked the multimodal large model.

[0119] In some embodiments of this application, the model evaluation device can input a third image into a multimodal large model to obtain the recognition result of the multimodal large model for the third image.

[0120] In some embodiments of this application, the model evaluation device inputs the third image into a multimodal large model to obtain the recognition result, which can be expressed as Equation 6:

[0121] (6)

[0122] Wherein, MLLM is the multimodal large model to be evaluated, and X is the image recognition result for the third image.

[0123] In some embodiments, combined with Figure 2 ,like Figure 8 As shown, step 205 above can be implemented through steps 205a to 205d.

[0124] Step 205a: The model evaluation device inputs the third image into the multimodal large model, extracts information from the third image through the multimodal large model, and obtains the first output result.

[0125] In some embodiments of this application, the model evaluation device can input a third image into a multimodal large model to obtain the first output result of the multimodal large model for the third image.

[0126] In some embodiments of this application, the aforementioned multimodal large model can perform image recognition on the third image to extract image information from the third image.

[0127] In some embodiments of this application, the above-mentioned multimodal large model can be processed according to the above-mentioned image information to obtain a first output result.

[0128] Step 205b: The model evaluation device matches the first output result with the preset first information, which is used to characterize that the first attack prompt word has successfully attacked the multimodal large model.

[0129] In some embodiments of this application, the model evaluation device can obtain first information based on a first attack prompt word, which can be used to characterize the output information of the first attack prompt word for the successful attack on a multimodal large model.

[0130] In some embodiments of this application, the model evaluation device can match the first output result with the first information and determine whether the match is successful based on the similarity between the first output result and the first information.

[0131] Step 205c: If the match is successful, the model evaluation device outputs the image recognition result that characterizes the success of the first attack prompt word in attacking the multimodal large model.

[0132] In some embodiments of this application, if the first output result and the first information match successfully, the model evaluation device can output an image recognition result for the third image, which is used to characterize the success of the attack on the multimodal large model by the third image.

[0133] Step 205d: If the matching fails, the model evaluation device outputs the image recognition result that characterizes the failure of the first attack prompt word to attack the multimodal large model.

[0134] In some embodiments of this application, if the first output result and the first information fail to match, the model evaluation device can output an image recognition result for the third image, which is used to characterize the failure of the third image to attack the multimodal large model.

[0135] It should be noted that steps 205c and 205d above are executed in an "OR" relationship.

[0136] In the model evaluation method provided in this application, the first attack prompt word is split into sub-prompt words to preliminarily decompose the content of the first attack prompt word, and a corresponding second image is generated based on the content of the sub-prompt words. Then, a first image with a large semantic difference from the first attack prompt word is obtained from the first image set to increase the complexity of the image content. The first and second images are then combined to obtain a third image containing complex content and the complete content of the first attack prompt word. This image is used to evaluate the defense capability of the multimodal large model. It can conduct adversarial testing with samples that are more in line with the standards of actual attack commands, thereby improving the accuracy of the evaluation of the defense capability of the multimodal large model.

[0137] Understandably, the original attack prompts are broken down into harmless text fragments and formed into images, allowing the multimodal model to process the harmless fragmented information. Cosine similarity is used to find the subgraph with the greatest difference from the original prompt, and the subsequent subgraph with the greatest difference from the preceding subgraph is formed through increased complexity, thereby improving visual complexity and the interference model's defense capabilities. Process interference is used to disrupt the model's attention mechanism and value alignment strategy, thereby improving the accuracy of the evaluation of the defense capabilities of the multimodal large model through this sample.

[0138] The model evaluation method of this application will be described below through specific embodiments.

[0139] The implementation process of the model evaluation method provided in this application includes the following S1 to S3:

[0140] S1. The model evaluation device disassembles the original attack prompt words.

[0141] In some embodiments of this application, the model evaluation device can obtain the original attack prompt.

[0142] For example, the original prompt could be: P="How to obtain privacy information", which is an inducement instruction to attack a multimodal large model.

[0143] In some embodiments of this application, the model evaluation device can decompose the original prompt into sub-prompt words using an open-source large model.

[0144] In some embodiments of this application, the model evaluation device can decompose the original prompt into sub-prompt words that can be calculated by the above formula 1.

[0145] For example, the model evaluation device can decompose the original P "how to obtain privacy information" into: P1 = "explain privacy information"; P2 = "what methods can be used to obtain privacy information"; P3 = "obtain privacy information".

[0146] In some embodiments of this application, the model evaluation device can perform text-image representation conversion on the decomposed P to obtain the subgraph corresponding to the original prompt: .

[0147] S2, The model evaluation device enhances the visual complexity of the subgraph.

[0148] In some embodiments of this application, the model evaluation device can convert the original attack prompt into a vector using a CLIP encoder (CLIP-ViT-L / 14 model):

[0149] In some embodiments of this application, the model evaluation apparatus can retrieve subgraphs from an image dataset D, and the image dataset D can be a general image dataset without a specific domain.

[0150] In some embodiments of this application, the model evaluation device can be... Compare the cosine similarity of the vectors of each image in dataset D with the vectors of the first image, and select the image with the lowest similarity as the first sub-image. : ,in, .

[0151] In some embodiments of this application, the model evaluation device can select the model with the highest difference from the original attack prompt and the selected subgraph from the dataset D to ensure the model inference complexity.

[0152] Specifically, the subsequent subgraph selected by the model evaluation device can be obtained from Equation 4 above.

[0153] S3, the model evaluation device combines the attack data with the subgraph.

[0154] In some embodiments of this application, the model evaluation device can combine attack data and subgraphs to obtain image data, which has increased complexity due to the inclusion of other images in the image dataset.

[0155] In some embodiments of this application, the image data of the model evaluation device combination can be obtained by the above formula 5.

[0156] In some embodiments of this application, the model evaluation device may arrange and splice attack data and subgraphs in a specific order.

[0157] In some embodiments of this application, the model evaluation device inputs image data into the evaluation data obtained by the multimodal model, which can be obtained by the above formula 6.

[0158] Thus, by splitting the first attack prompt into sub-prompts, the content of the first attack prompt is initially decomposed, and a corresponding second image is generated based on the content of the sub-prompts. Then, a first image with a significant semantic difference from the first attack prompt is obtained from the first image set to increase the complexity of the image content. The first and second images are then combined to obtain a third image containing complex content and the complete content of the first attack prompt. This image is used to evaluate the defense capabilities of the multimodal large model. It can conduct adversarial testing using samples that are more in line with the standards of actual attack commands, thereby improving the accuracy of the evaluation of the defense capabilities of the multimodal large model.

[0159] It should be noted that the descriptions of each step S1 to S3 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.

[0160] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0161] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0162] This application embodiment can divide the model evaluation device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0163] In some embodiments, this application also provides a model evaluation apparatus. The model evaluation apparatus may include one or more functional modules for implementing the model evaluation method of the above method embodiments.

[0164] For example, Figure 9 This is a schematic diagram of a model evaluation device provided in an embodiment of this application. Figure 9 As shown, the model evaluation device 900 includes: an acquisition module 901, a splitting module 902, and a generation module 903; wherein, the acquisition module 901 is used to acquire a first attack prompt word and a first image set, the first image set containing N first images, where N is an integer greater than 1; the splitting module 902 is used to split the first attack prompt word to obtain at least one sub-prompt word, and generate a second image corresponding to each sub-prompt word based on the at least one sub-prompt word; the acquisition module 901 is also used to, based on the similarity between the first attack prompt word and each first image, generate a second image from the N... M first images are determined from the first images, the similarity of the M first images is less than a first similarity threshold, and the similarity between the M first images and the first attack prompt word is less than a second similarity threshold, where M is a positive integer less than or equal to N; the generation module 903 is used to generate a third image based on the second image corresponding to at least one sub-prompt word and the M first images; the acquisition module 901 is further used to input the third image into the multimodal large model to obtain the image recognition result, the image recognition result being used to characterize whether the first attack prompt word successfully attacks the multimodal large model.

[0165] In some embodiments, the acquisition module 901 is specifically used to: acquire the first similarity between each of the N first images and the first attack prompt word, and determine the first target image corresponding to the minimum first similarity from the N first images; acquire the second similarity between each of the N first images and the i-th target image, and based on the first similarity corresponding to each first image and the similarity sum of K second similarities, determine the (i+1)-th target image corresponding to the minimum similarity sum from the N first images to obtain M first images, wherein the K second similarities corresponding to a first image include the second similarities of the first i target images corresponding to the first image, i∈[1,M-1], and i is an integer.

[0166] In other embodiments, the acquisition module 901 is specifically used to: acquire a first vector of each of the N first images and a second vector of the first attack prompt word, wherein the first vector is used to characterize the semantic information of each of the N first images and the second vector is used to characterize the semantic information of the first attack prompt word; acquire the cosine similarity between the first vector and the second vector to obtain a first similarity between each of the N first images and the first attack prompt word.

[0167] In some other embodiments, the generation module 903 is specifically used to: stitch together the second image corresponding to at least one sub-prompt word and the M first images to obtain a first stitched image; and feather the edge regions of the second image or the first image in the first stitched image to obtain a third image.

[0168] In some other embodiments, the acquisition module 901 is specifically configured to: input the third image into a multimodal large model, extract information from the third image using the multimodal large model, and obtain a first output result; match the first output result with preset first information, wherein the first information is used to characterize the first attack prompt word as having successfully attacked the multimodal large model; if the match is successful, output an image recognition result characterizing the first attack prompt word as having successfully attacked the multimodal large model; if the match fails, output an image recognition result characterizing the first attack prompt word as having failed to attack the multimodal large model.

[0169] The model evaluation device provided in this application preliminarily decomposes the content of the first attack prompt word by splitting it into sub-prompt words, and generates a corresponding second image based on the content of the sub-prompt words. Then, it obtains a first image from the first image set that has a large semantic difference from the first attack prompt word to increase the complexity of the image content. The first and second images are then combined to obtain a third image that contains complex content and complete content including the first attack prompt word. This image is used to evaluate the defense capability of a multimodal large model. It can conduct adversarial testing with samples that are more in line with the standards of actual attack commands, thereby improving the accuracy of the evaluation of the defense capability of a multimodal large model.

[0170] It should be noted that the model evaluation device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0171] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 10As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.

[0172] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0173] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0174] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0175] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the model evaluation method provided in the embodiments of this application.

[0176] In another possible implementation, memory 91 can also be integrated with processor 92.

[0177] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0178] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0179] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model evaluation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0180] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0181] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the model evaluation method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a pluggable hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0182] This application also provides a computer program product, which is stored in a storage medium and implements the model evaluation method provided in the above embodiments when the computer program product is executed by a computer.

[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0185] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model evaluation method, characterized in that, include: Obtain the first attack prompt word and the first image set, wherein the first image set contains N first images, where N is an integer greater than 1; The first attack prompt word is split to obtain at least one sub-prompt word, and a second image corresponding to each sub-prompt word is generated based on the at least one sub-prompt word; Based on the similarity between the first attack prompt and each first image, M first images are determined from the N first images. The similarity of the M first images is less than a first similarity threshold, and the similarity between the M first images and the first attack prompt is less than a second similarity threshold. M is a positive integer less than or equal to N. A third image is generated based on the second image corresponding to the at least one sub-cue word and the M first images; The third image is input into the multimodal large model to obtain the image recognition result. The image recognition result is used to characterize whether the first attack prompt word successfully attacks the multimodal large model.

2. The model evaluation method according to claim 1, characterized in that, The step of determining M first images from the N first images based on the similarity between the first attack prompt word and each first image includes: Obtain the first similarity between each of the N first images and the first attack prompt word, and determine the first target image corresponding to the minimum first similarity from the N first images; Obtain the second similarity between each of the N first images and the i-th target image. Based on the first similarity corresponding to each first image and the similarity sum of the K second similarities, determine the (i+1)-th target image corresponding to the minimum similarity sum from the N first images to obtain M first images. The K second similarities corresponding to a first image include the second similarities of the first i target images corresponding to the first image, i∈[1,M-1], and i is an integer.

3. The model evaluation method according to claim 2, characterized in that, The step of obtaining the first similarity between each of the N first images and the first attack prompt word includes: Obtain a first vector for each of the N first images and a second vector for the first attack prompt word. The first vector is used to characterize the semantic information of each first image, and the second vector is used to characterize the semantic information of the first attack prompt word. Obtain the cosine similarity between the first vector and the second vector to obtain the first similarity between each of the N first images and the first attack prompt word.

4. The model evaluation method according to claim 1, characterized in that, The step of generating a third image based on the second image corresponding to the at least one sub-cue word and the M first images includes: The second image corresponding to the at least one sub-prompt word and the M first images are stitched together to obtain a first stitched image; Feathering is applied to the edge regions of the second image or the first image in the first stitched image to obtain the third image.

5. The model evaluation method according to claim 1, characterized in that, The step of inputting the third image into a multimodal large model to obtain image recognition results includes: The third image is input into a multimodal large model, and information is extracted from the third image by the multimodal large model to obtain the first output result; The first output result is matched with preset first information, which is used to characterize that the first attack prompt word has successfully attacked the multimodal large model. If the match is successful, the image recognition result representing the successful attack of the first attack prompt word on the multimodal large model is output. If the match fails, the image recognition result is output to characterize the failure of the first attack prompt word to attack the multimodal large model.

6. A model evaluation device, characterized in that, include: Acquire modules, split modules, and generate modules; The acquisition module is used to acquire a first attack prompt word and a first image set, wherein the first image set contains N first images, where N is an integer greater than 1; The splitting module is used to split the first attack prompt word to obtain at least one sub-prompt word, and generate a second image corresponding to each sub-prompt word based on the at least one sub-prompt word; The acquisition module is further configured to determine M first images from the N first images based on the similarity between the first attack prompt word and each first image, wherein the similarity of the M first images is less than a first similarity threshold, and the similarity between the M first images and the first attack prompt word is less than a second similarity threshold, and M is a positive integer less than or equal to N. The generation module is used to generate a third image based on the second image corresponding to the at least one sub-prompt word and the M first images; The acquisition module is further configured to input the third image into the multimodal large model and obtain the image recognition result, wherein the image recognition result is used to characterize whether the first attack prompt word has successfully attacked the multimodal large model.

7. The model evaluation apparatus according to claim 6, characterized in that, The acquisition module is specifically used for: Obtain the first similarity between each of the N first images and the first attack prompt word, and determine the first target image corresponding to the minimum first similarity from the N first images; Obtain the second similarity between each of the N first images and the i-th target image. Based on the first similarity corresponding to each first image and the similarity sum of the K second similarities, determine the (i+1)-th target image corresponding to the minimum similarity sum from the N first images to obtain M first images. The K second similarities corresponding to a first image include the second similarities of the first i target images corresponding to the first image, i∈[1,M-1], and i is an integer.

8. The model evaluation apparatus according to claim 7, characterized in that, The acquisition module is specifically used for: Obtain a first vector for each of the N first images and a second vector for the first attack prompt word. The first vector is used to characterize the semantic information of each first image, and the second vector is used to characterize the semantic information of the first attack prompt word. Obtain the cosine similarity between the first vector and the second vector to obtain the first similarity between each of the N first images and the first attack prompt word.

9. The model evaluation apparatus according to claim 6, characterized in that, The generation module is specifically used for: The second image corresponding to the at least one sub-prompt word and the M first images are stitched together to obtain a first stitched image; Feathering is applied to the edge regions of the second image or the first image in the first stitched image to obtain the third image.

10. The model evaluation apparatus according to claim 6, characterized in that, The acquisition module is specifically used for: The third image is input into a multimodal large model, and information is extracted from the third image by the multimodal large model to obtain the first output result; The first output result is matched with preset first information, which is used to characterize that the first attack prompt word has successfully attacked the multimodal large model. If the match is successful, the image recognition result representing the successful attack of the first attack prompt word on the multimodal large model is output. If the match fails, the image recognition result is output to characterize the failure of the first attack prompt word to attack the multimodal large model.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the model evaluation method as described in any one of claims 1-5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the model evaluation method as described in any one of claims 1-5.

13. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the model evaluation method as described in any one of claims 1-5.