Risk identification method, device, equipment, medium and program product
Patent Information
- Application Number
- CN202611114396.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-24
- Publication Date
- 2026-10-30
AI Technical Summary
[0004]然而，由于这种外挂式大模型与多模态大模型彼此独立，无法深入感知模型内部对多模态数据的实际处理逻辑与特征融合方式，导致在面对具有高度针对性的跨模态语义级攻击、隐式指令注入等攻击手段时，感知能力显著不足，导致对多模态大模型进行攻击识别的准确性较低
[0020]第六方面，本申请实施例提供了一种芯片，该芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第一方面所述的方法。
Smart Images
Figure CN123013087A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a risk identification method, apparatus, device, medium, and program product. Background Technology
[0002] With the widespread application of multimodal large models, the security threats they face are becoming increasingly complex. Therefore, it is necessary to establish a defense mechanism before using multimodal large models to prevent potential attacks in advance.
[0003] Existing technologies typically employ a "large model fence" strategy for protection. This involves deploying a protective layer independent of the large multimodal model outside of it. When multimodal input data enters, this protective layer will first review and intercept it, thereby ensuring the security of the data input to the large model to a certain extent.
[0004] However, because this plug-in large model is independent of the multimodal large model, it is impossible to deeply perceive the actual processing logic and feature fusion method of multimodal data inside the model. As a result, the perception capability is significantly insufficient when facing highly targeted cross-modal semantic level attacks, implicit instruction injection and other attack methods, resulting in low accuracy in attack identification of multimodal large models. Summary of the Invention
[0005] This application provides a risk identification method, apparatus, device, medium, and program product to improve the accuracy of attack identification for multimodal large models.
[0006] In a first aspect, embodiments of this application provide a risk identification method, the method comprising: obtaining first feature information based on first text and a first image through at least one conversion layer of a multimodal large model, wherein the first feature information includes: image features of the first image output by each conversion layer, text features of the first text, and text description features of the first image; determining a first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer, wherein the image features and text description features output by the first conversion layer have the highest similarity; obtaining a first index of the first conversion layer based on the image features and text description features output by the first conversion layer, and obtaining a second index of the first conversion layer based on the image features and text features output by the first conversion layer, wherein the first index is used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index is used to characterize the semantic deviation between the first image and the first text; and obtaining a first determination result based on the first index and the second index, wherein the first determination result is used to characterize whether the first text and the first image have an attack risk.
[0007] The technical solution provided in this application brings at least the following beneficial effects: First feature information is obtained based on first text and first image through at least one conversion layer of a multimodal large model; a first conversion layer is determined from the at least one conversion layer based on the first feature information output by each conversion layer; a first index and a second index of the first conversion layer are obtained based on the first feature information output by the first conversion layer; and a first judgment result is obtained based on the first index and the second index. In this solution, first feature information of the first text and first image is obtained through each conversion layer in at least one conversion layer, and the similarity of the processed information of the image and text by each conversion layer is analyzed based on the first feature information to obtain the optimal alignment layer (i.e., the first conversion layer) with the highest quality inspection similarity between the processed text and image. Then, the attack power of the input image and input text is judged by the first similarity index of the image and text output by the first conversion layer and the second similarity index of the image and image description. Since the optimal alignment layer is selected based on the output content, and the first and second similarity indices of the optimal alignment layer are used, the attack power judgment of the input text and input image can be based on the most accurate conversion layer, thereby improving the accuracy of attack identification by the multimodal large model.
[0008] One possible implementation is that, based on the first text and the first image, the first feature information is obtained through at least one conversion layer of the multimodal large model, including: obtaining a text description of the first image based on the first image; and inputting the first text, the first image, and the text description of the first image into each of the at least one conversion layer to obtain the first feature information.
[0009] Another possible implementation, which determines the first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer, includes: obtaining a first similarity of each conversion layer based on the image features and text features output by each conversion layer; obtaining the maximum value of the first similarity of each conversion layer; and determining the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
[0010] Another possible implementation is to obtain the second index of the first conversion layer based on the image features and text features output by the first conversion layer, which includes: obtaining the covariance similarity of the image features and text features output by the first conversion layer, and obtaining the second index based on the covariance similarity.
[0011] Another possible implementation is that, based on the first index and the second index, the first determination result is obtained by: obtaining a first difference between the second index and the first index; if the first difference is greater than or equal to a first preset threshold, the first determination result is that the first text and the first image are risky; if the first difference is less than the first preset threshold, the first determination result is that the first text and the first image are not risky.
[0012] Secondly, embodiments of this application provide a risk identification device, comprising: an acquisition module; the acquisition module being configured to acquire first feature information based on first text and a first image through at least one conversion layer of a multimodal large model, the first feature information including: image features of the first image output by each conversion layer, text features of the first text, and text description features of the first image; the acquisition module being further configured to determine a first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer, wherein the image features and text description features output by the first conversion layer have the highest similarity; the acquisition module being further configured to acquire a first index of the first conversion layer based on the image features and text description features output by the first conversion layer, and acquire a second index of the first conversion layer based on the image features and text features output by the first conversion layer, the first index being used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index being used to characterize the semantic deviation between the first image and the first text; the acquisition module being further configured to acquire a first determination result based on the first index and the second index, the first determination result being used to characterize whether the first text and the first image have an attack risk.
[0013] One possible implementation is that the acquisition module is specifically used to: acquire a text description of the first image based on the first image; input the first text, the first image, and the text description of the first image into each of the at least one conversion layer to acquire first feature information.
[0014] Another possible implementation, the above acquisition module is specifically used to: obtain the first similarity of each conversion layer based on the image features and text features output by each conversion layer; obtain the maximum value of the first similarity of each conversion layer; and determine the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
[0015] Another possible implementation is that the acquisition module described above is specifically used to: acquire the covariance similarity between the image features and text features output by the first conversion layer, and obtain the second index based on the covariance similarity.
[0016] Another possible implementation is that the above-mentioned acquisition module is specifically used to: acquire the first difference between the second index and the first index; if the first difference is greater than or equal to a first preset threshold, then acquire the first determination result that the first text and the first image are risky; if the first difference is less than the first preset threshold, then acquire the first determination result that the first text and the first image are not risky.
[0017] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.
[0018] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.
[0019] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.
[0020] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.
[0021] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description
[0022] Figure 1 A schematic diagram of the network architecture for an application of a risk identification method provided in this application embodiment;
[0023] Figure 2 A flowchart illustrating a risk identification method provided in an embodiment of this application;
[0024] Figure 3 A flowchart illustrating a risk identification method provided in an embodiment of this application;
[0025] Figure 4 A schematic diagram of the optimal alignment layer for a risk identification method provided in an embodiment of this application;
[0026] Figure 5 A flowchart illustrating a risk identification method provided in an embodiment of this application;
[0027] Figure 6A flowchart illustrating yet another risk identification method provided in this application embodiment;
[0028] Figure 7 A flowchart illustrating a risk identification method provided in an embodiment of this application;
[0029] Figure 8 This is a schematic diagram of the structure of a risk identification device provided in an embodiment of this application;
[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] The risk identification method, apparatus, equipment, media, and program products provided in this application will now be described in detail with reference to the accompanying drawings.
[0032] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0033] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0034] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."
[0035] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0036] The risk identification method, apparatus, device, medium, and program product provided in this application embodiment can be applied to scenarios involving attack identification of multimodal large models.
[0037] With the widespread application of multimodal large models, the security threats they face are becoming increasingly complex. Therefore, it is necessary to establish a defense mechanism before using multimodal large models to prevent potential attacks in advance.
[0038] Existing technologies typically employ a "large model fence" strategy for protection. This involves deploying a protective layer independent of the large multimodal model outside of it. When multimodal input data enters, this protective layer will first review and intercept it, thereby ensuring the security of the data input to the large model to a certain extent.
[0039] However, because this plug-in large model is independent of the multimodal large model, it cannot deeply perceive the actual processing logic and feature fusion methods of multimodal data within the model. This results in a significant deficiency in perception when facing highly targeted cross-modal semantic attacks, implicit instruction injection, and other attack methods, leading to low accuracy in attack identification of multimodal large models.
[0040] Existing large-scale model security defenses mainly rely on external large-scale model fences, resulting in defense lag, response delays, and an inability to identify cross-modal semantic-level jailbreak behaviors in real time. They also lack the ability to detect new types of attacks such as multi-step collaboration and implicit command injection. Furthermore, fence rules rely on manual definition, have poor generalization, and are difficult to cover dynamically evolving attack patterns.
[0041] This application provides an adversarial attack detection and mitigation method that does not require retraining and can be directly embedded into mainstream multimodal Transformer architectures. It is applicable to scenarios involving jailbreak attacks and cross-modal adversarial attack protection for large multimodal models.
[0042] This paper presents a lightweight, embeddable, dynamically adaptable, end-to-end cross-modal adversarial attack defense method, which achieves accurate detection and real-time mitigation of multimodal adversarial attacks without modifying the original model weights or affecting normal inference performance.
[0043] By traversing all layers of the Transformer Block, the optimal alignment layer for text and image input is found as the benchmark anchor point for risk assessment. The alignment degree between multimodal semantics is quantified by activating the ASI cheapness index to achieve the detection of malicious attacks.
[0044] To address the aforementioned technical problems, embodiments of this application provide a risk identification method, apparatus, device, medium, and program product. The method acquires first feature information of first text and first image through each of at least one conversion layer. Based on the first feature information, it analyzes the similarity of the processed information of the image and text by each conversion layer to obtain the optimal alignment layer (i.e., the first conversion layer) with the highest quality inspection similarity between the processed text and image. Then, it uses the first similarity index of the image and text output by the first conversion layer and the second similarity index of the image and image description to determine the attack power of the input image and input text. Since the optimal alignment layer is selected based on the output content, and the first and second similarity indices of the optimal alignment layer are used, the attack power determination of the input text and input image can be based on the most accurate conversion layer, thereby improving the accuracy of attack identification using a multimodal large model.
[0045] The risk identification methods, apparatus, devices, media, and program products provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0046] Figure 1 The diagram illustrates a network architecture for an application of a risk identification method provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a risk identification device 101 and a terminal device 102. The risk identification device 101 and the terminal device 102 are interconnected.
[0047] In some embodiments, the risk identification device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster comprising multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the risk identification device 101. Figure 1 The risk identification device 101 is used as an example of a single server.
[0048] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.
[0049] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0050] See Figure 2 This is a flowchart illustrating a risk identification method provided in an embodiment of this application. Figure 2 As shown, the risk identification method provided in this application embodiment can be implemented by the above-mentioned risk identification device, specifically including the following steps 201 to 204.
[0051] Step 201: The risk identification device obtains first feature information based on the first text and the first image through at least one conversion layer of the multimodal large model.
[0052] In some embodiments of this application, the first feature information mentioned above includes: image features of the first image output by each conversion layer, text features of the first text, and text description features of the first image.
[0053] In some embodiments of this application, the risk identification device can extract features of the first text and the first image at the input layer.
[0054] In some embodiments of this application, the risk identification device can input the first text and the first image into each Transformer and extract the cross-modal hidden state of each layer.
[0055] In some embodiments of this application, the risk identification device can obtain image features of the first image, text features of the first text, and text description features of the first image from the aforementioned cross-modal hiding state.
[0056] In some embodiments, combined with Figure 2 ,like Figure 3 As shown, step 201 above can be implemented through steps 201a and 201b.
[0057] Step 201a: The risk identification device obtains a text description of the first image based on the first image.
[0058] In some embodiments of this application, the text description of the first image described above is used to characterize the content contained in the first image.
[0059] In some embodiments of this application, the risk identification device can preprocess the first image using models such as Large Language and Vision Assistant (LLaVA) to generate a text description of the image.
[0060] Step 201b: The risk identification device inputs the first text, the first image, and the text description of the first image into each of at least one conversion layer to obtain the first feature information.
[0061] In some embodiments of this application, the first The image features of the layer are: d is the feature dimension of the Transformer, and the average pooled hidden state of all image tokens in this layer is taken.
[0062] In some embodiments of this application, the first The text features of the layer are: d is the feature dimension of the Transformer, which is the average pooled hidden state of all text tokens in this layer.
[0063] In some embodiments of this application, the first The descriptive features of a layer are: d is the feature dimension of the Transformer, and the average pooled hidden state of all image description tokens in this layer is taken.
[0064] In some embodiments of this application, the above-mentioned first... The layer can be any one of the entire conversion layers, and each of the entire conversion layers can be converted through the above-mentioned... Feature extraction is performed using layer-based feature extraction methods.
[0065] In this way, by inputting the original image, text description, and extracted feature information into the transformation layer, the direct output results of different transformation layers can be fully obtained, so as to perform the analysis of the optimal alignment layer based on the output results of each transformation layer.
[0066] Step 202: The risk identification device determines the first conversion layer from at least one conversion layer based on the image features and text description features output by each conversion layer.
[0067] In some embodiments of this application, the image features and text description features output by the first conversion layer have the highest similarity.
[0068] In some embodiments of this application, the risk identification device can find the optimal semantic alignment layer in the Transformer architecture based on the multimodal input of the first text and the first image, and use it as the benchmark for subsequent ASI offset index evaluation, thereby avoiding the deviation caused by using a fixed layer as the benchmark in different input semantics.
[0069] In some embodiments of this application, the degree of semantic alignment between the output of the first text corresponding to the above-mentioned optimal semantic alignment layer and the output of the first image corresponding to the above-mentioned optimal semantic alignment layer is the largest among all conversion layers.
[0070] It should be noted that when performing semantic alignment in a multimodal large model, the position of the optimal alignment layer is related to the complexity of the input task. For simple multimodal input tasks, the model can quickly complete the semantic alignment of cross-modal features, and the optimal semantic alignment layer is usually in a relatively early transformation layer in the multimodal large model. Conversely, for complex multimodal input tasks, the model needs to perform deeper processing, and its semantic alignment will be in a relatively late transformation layer in the multimodal large model.
[0071] It should be further noted that the optimal semantic alignment layer in a multimodal large model is not fixed, but is related to the complexity of the input task. Therefore, the traditional approach of using a fixed alignment layer is difficult to match different input tasks, making it difficult to obtain an accurate optimal alignment layer.
[0072] For example, such as Figure 4 The figure shows a schematic diagram of the optimal alignment layer. The horizontal axis represents the number of transformation layers in the multimodal large model, and the vertical axis represents the feature modulus of the text representation or the image representation. The smaller the difference between the feature modulus of the two, the closer the semantics of the alignment layer are to the alignment. As shown in the figure, when the feature modulus of the text representation and the image representation are equal, the corresponding layer is the optimal alignment layer.
[0073] In some embodiments, combined with Figure 2 ,like Figure 5 As shown, step 202 above can be implemented through steps 202a and 202b.
[0074] Step 202a: The risk identification device obtains the first similarity of each conversion layer based on the image features and text features output by each conversion layer.
[0075] In some embodiments of this application, the risk identification device can calculate the semantic similarity between image features and image description features for each layer using cosine similarity: ,in, Indicates the first The first similarity between the image features and image description features of the layer. Indicates the first Image features of the layer, Indicates the first Descriptive features of layers.
[0076] Step 202b: The risk identification device obtains the maximum value of the first similarity of each conversion layer and determines the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
[0077] In some embodiments of this application, the first similarity value ranges from -1 to 1, with a larger value indicating a higher degree of semantic alignment. The layer with the largest value is the optimal alignment layer. .
[0078] This can solve the problems of inconsistent optimal alignment layers and unsuitable detection thresholds for different inputs in existing alignment defense technologies.
[0079] Step 203: The risk identification device obtains the first index of the first conversion layer based on the image features and text description features output by the first conversion layer, and obtains the second index of the first conversion layer based on the image features and text features output by the first conversion layer.
[0080] In some embodiments of this application, the first index is used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index is used to characterize the semantic deviation between the first image and the first text.
[0081] In some embodiments of this application, the risk identification device can determine whether there is malicious attack content in the multimodal input content by quantifying the degree of semantic offset.
[0082] In some embodiments of this application, the risk identification device can obtain a first index (i.e., a benchmark ASI) when the image input and description are optimally aligned. .
[0083] In some embodiments of this application, the first index is the minimum ASI value under no-attack conditions.
[0084] In some embodiments of this application, the risk identification device can obtain a second index (i.e., the actual ASI offset of the image / text).
[0085] In some embodiments of this application, the risk identification device can use input image features and text features to calculate the actual ASI at the optimal alignment layer.
[0086] In some embodiments of this application, the risk identification device can obtain the covariance of image features and text features, and obtain the actual ASI based on the covariance.
[0087] In some embodiments, the step 203 above, "obtaining the second index of the first conversion layer based on the image features and text features output by the first conversion layer", can be specifically implemented through the following step 203a.
[0088] Step 203a: The risk identification device obtains the covariance similarity of the image features and text features output by the first conversion layer, and obtains the second index based on the covariance similarity.
[0089] In some embodiments of this application, the risk identification device can use input image features and text features to obtain the covariance of image features and text features at the optimal alignment layer: ,in, Image features representing the optimal alignment layer. The image text description features representing the optimal alignment layer.
[0090] In some embodiments of this application, the covariance of the above-mentioned image features and text features can represent the similarity between the image features and text features.
[0091] In some embodiments of this application, the risk identification device can obtain a second index based on covariance similarity: ,in, Indicates the first The covariance of image features and text features output by the layer. This represents the ASI of the optimal alignment layer.
[0092] Step 204: The risk identification device obtains a first judgment result based on the first index and the aforementioned second index.
[0093] In some embodiments of this application, the first determination result is used to characterize whether the first text and the first image have an attack risk.
[0094] In some embodiments of this application, the risk identification device can obtain the degree of difference between the first index and the second index, and obtain a first determination result based on the degree of difference.
[0095] In some embodiments of this application, the risk identification device can characterize the first text and the first image as having an attack risk by means of the first determination result when the difference between the first index and the second index is greater than or equal to the first threshold.
[0096] In some embodiments of this application, the risk identification device can characterize the first text and the first image as not having an attack risk by means of the first determination result when the difference between the first index and the second index is less than a first threshold.
[0097] For example, such as Figure 6The diagram illustrates the risk identification process. When processing normal input, the multimodal model achieves semantic alignment between image and text representations at a certain layer. However, with adversarial input, this alignment is broken, manifesting as "activation shift." After the input layer of the multimodal model receives image and text input, the processing layer uses the LLaVA model to generate a text description of the input image, achieving image-to-description generation. Then, in the feature extraction layer, image features, text features, and image description text features are obtained through transformation layers h0, h1…h_(n-1), and h_n. The hidden state of the last token is extracted at each layer, sequentially completing cross-modal representation extraction and layer-by-layer alignment analysis to find the optimal alignment layer. Finally, the extracted features are analyzed using the ASI activation shift index to quantify the degree of feature shift. This is compared with a preset threshold. If the ASI is greater than or equal to the preset threshold, an adversarial attack is detected, and an interception is output. If the ASI is less than the preset threshold, the input image and text are considered normal input, and the model-generated result is directly output.
[0098] In some embodiments, combined with Figure 2 ,like Figure 7 As shown, step 204 above can be implemented through steps 204a to 204c.
[0099] Step 204a: The risk identification device obtains the first difference between the second index and the first index.
[0100] In some embodiments of this application, the risk identification device can obtain the difference between the second index and the first index, i.e., the ASI offset. The first difference represents the degree of semantic alignment deviation. The larger the first difference is, the higher the risk level of the first image and the first text.
[0101] In some embodiments of this application, the first difference can be obtained based on the difference between the second exponent and the first exponent: ,in, Indicates the second index, This indicates the first index.
[0102] In some embodiments of this application, the larger the first difference is, the higher the degree of semantic alignment deviation caused by the attack, i.e., the higher the risk level.
[0103] Step 204b: If the first difference is greater than or equal to the first preset threshold, the risk identification device obtains the first judgment result that the first text and the first image are risky.
[0104] In some embodiments of this application, if the first difference is greater than or equal to the first preset threshold, the risk identification device determines the first text and the first image as having risk.
[0105] Step 204c: If the first difference is less than the first preset threshold, the risk identification device obtains the first judgment result as the first text and the first image being risk-free.
[0106] In some embodiments of this application, if the first difference is less than the first preset threshold, the risk identification device determines the first text and the first image as risk-free.
[0107] It should be noted that the execution relationship between steps 204b and 204c is "OR", that is, either step 204b or step 204c is executed.
[0108] In the risk identification method provided in this application, first feature information of the first text and the first image is obtained through each of at least one conversion layer. Based on the first feature information, the similarity of the processed information of the image and text by each conversion layer is analyzed to obtain the optimal alignment layer (i.e., the first conversion layer) with the highest quality inspection similarity between the processed text and the image. Then, the attack power of the input image and input text is judged by the first similarity index of the image and text output by the first conversion layer and the second similarity index of the image and the image description. Since the optimal alignment layer is selected by the output content, and the first similarity index and the second similarity index of the optimal alignment layer are used, the attack power judgment of the input text and input image can be based on the most accurate conversion layer, thereby improving the accuracy of attack identification in multimodal large models.
[0109] Understandably, the activation offset index quantitative evaluation based on the dynamic multimodal optimal semantic alignment layer solves the problem of poor generalization of existing external large model security fences from an endogenous perspective, and will be applied to AI security governance scenarios for the discovery and protection of multimodal AI security risks.
[0110] The risk identification method of this application will be described below through specific embodiments.
[0111] The implementation process of the risk identification method provided in this application includes the following steps S1 to S4:
[0112] By traversing all layers of the Transformer Block, the optimal alignment layer for text and image input is found as the benchmark anchor point for risk assessment. The alignment degree between multimodal semantics is quantified by activating the ASI cheapness index to achieve the detection of malicious attacks.
[0113] S1. The risk identification device extracts features from text input and image input at the input layer.
[0114] In some embodiments of this application, the risk identification device can acquire text input as Q and image input as I.
[0115] In some embodiments of this application, the risk identification device can preprocess image I using models such as LLaVA to generate a text description C for the image.
[0116] S2. The risk identification device inputs the text Q, image I, and image description C into the Transformer, traverses all L layers, and extracts the cross-modal hidden state of each layer.
[0117] In some embodiments of this application, , No. The image features of the layer are represented as d is the feature dimension of the Transformer, and the average pooled hidden state of all image tokens in this layer is taken.
[0118] In some embodiments of this application, the first The text features of the layer are d is the feature dimension of the Transformer, which is the average pooled hidden state of all text tokens in this layer.
[0119] In some embodiments of this application, the first The descriptive characteristics of a layer are: d is the feature dimension of the Transformer, and the average pooled hidden state of all image description tokens in this layer is taken.
[0120] S3. The risk identification device searches for the optimal alignment layer for multimodal analysis.
[0121] In some embodiments of this application, the risk identification device can find the optimal semantic alignment layer in the Transformer architecture for each text + image multimodal input pair, and use it as the benchmark for subsequent ASI offset index evaluation, thus avoiding the bias caused by using a fixed layer as the benchmark in different input semantics.
[0122] In some embodiments of this application, when performing semantic alignment, the position of the optimal alignment layer in a multimodal large model is related to the complexity of the input task. For simple multimodal input tasks, the model can quickly complete the semantic alignment of cross-modal features, and the optimal semantic alignment layer is usually in a relatively early transformation layer in the multimodal large model. Conversely, for complex multimodal input tasks, the model needs to perform deeper processing, and its semantic alignment will be in a relatively late transformation layer in the multimodal large model.
[0123] In some embodiments of this application, the risk identification device can calculate the semantic similarity between image features and image description features for each conversion layer using cosine similarity: , .
[0124] In some embodiments of this application, the semantic similarity value ranges from -1 to 1, with a larger value indicating a higher degree of semantic alignment.
[0125] In some embodiments of this application, the risk identification device can find the layer with the highest semantic similarity as the optimal alignment layer: .
[0126] This can solve the problems of inconsistent optimal alignment layers and unsuitable detection thresholds for different inputs in existing alignment defense technologies.
[0127] S4. The risk identification device calculates the ASI activation offset index.
[0128] In some embodiments of this application, the risk identification device can determine whether the multimodal input content contains malicious attack content by quantifying the degree of semantic offset.
[0129] In some embodiments of this application, the risk identification device can obtain the baseline ASI as the minimum ASI value under attack-free conditions when the image input and description are optimally aligned: .
[0130] In some embodiments of this application, the risk identification device uses input image features and text features to calculate the actual ASI at the optimal alignment layer: , .
[0131] In some embodiments of this application, the risk identification device can obtain the ASI offset as follows: .
[0132] In some embodiments of this application, the larger the ASI offset, the higher the degree of semantic alignment deviation caused by the attack, i.e., the higher the risk level.
[0133] In some embodiments of this application, the risk identification device can... In the case of τ, the input image and input text are determined to be risky, where τ is the preset attack detection threshold.
[0134] Thus, by obtaining the first feature information of the first text and the first image through each of at least one conversion layer, and analyzing the similarity of the processed information of the image and text by each conversion layer based on the first feature information, the optimal alignment layer (i.e., the first conversion layer) with the highest quality inspection similarity of the processed text and image is obtained. Then, the attack power of the input image and input text is judged by the first similarity index of the image and text output by the first conversion layer and the second similarity index of the image and image description. Since the optimal alignment layer is selected by the output content, and the first and second similarity indices of the optimal alignment layer are used, the attack power judgment of the input text and input image can be based on the most accurate conversion layer, thereby improving the accuracy of attack identification by multimodal large models.
[0135] It should be noted that the descriptions of each step S1 to S4 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.
[0136] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.
[0137] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0138] This application embodiment can divide the risk identification device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0139] In some embodiments, this application also provides a risk identification device. This risk identification device may include one or more functional modules for implementing the risk identification method of the above method embodiments.
[0140] For example, Figure 8 This is a schematic diagram of a risk identification device provided in an embodiment of this application. Figure 8 As shown, the risk identification device 900 includes: an acquisition module 901.
[0141] The acquisition module 901 is used to acquire first feature information based on the first text and the first image through at least one conversion layer of the multimodal large model. The first feature information includes: image features of the first image output by each conversion layer, text features of the first text, and text description features of the first image.
[0142] The acquisition module 901 is further configured to determine a first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer, wherein the image features and text description features output by the first conversion layer have the highest similarity.
[0143] The aforementioned acquisition module 901 is further configured to acquire a first index of the first conversion layer based on the image features and text description features output by the first conversion layer, and to acquire a second index of the first conversion layer based on the image features and text features output by the first conversion layer. The first index is used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index is used to characterize the semantic deviation between the first image and the first text.
[0144] The aforementioned acquisition module 901 is further configured to acquire a first determination result based on the aforementioned first index and the aforementioned second index, wherein the aforementioned first determination result is used to characterize whether the aforementioned first text and first image have an attack risk.
[0145] In some embodiments, the acquisition module 901 is specifically used to: acquire a text description of the first image based on the first image; input the first text, the first image, and the text description of the first image into each of the at least one conversion layer to acquire first feature information.
[0146] In other embodiments, the acquisition module 901 is specifically used to: acquire the first similarity of each conversion layer based on the image features and text features output by each conversion layer; acquire the maximum value of the first similarity of each conversion layer; and determine the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
[0147] In some other embodiments, the acquisition module 901 is specifically used to: acquire the covariance similarity between the image features and text features output by the first conversion layer, and obtain the second index based on the covariance similarity.
[0148] In some other embodiments, the acquisition module 901 is specifically used to: acquire a first difference between the second index and the first index; if the first difference is greater than or equal to a first preset threshold, then acquire a first determination result that the first text and the first image are risky; if the first difference is less than the first preset threshold, then acquire a first determination result that the first text and the first image are not risky.
[0149] It should be noted that the risk identification device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0150] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 9 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.
[0151] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0152] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.
[0153] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0154] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the risk identification method provided in the embodiments of this application.
[0155] In another possible implementation, memory 91 can also be integrated with processor 92.
[0156] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0157] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.
[0158] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described risk identification method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0159] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0160] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the risk identification method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the service invocation device, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, flash card, etc., equipped on the service invocation device. Further, the readable storage medium can include both internal storage units of the service invocation device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the service invocation device. The readable storage medium can also be used to temporarily store data that has been output or will be output.
[0161] This application also provides a computer program product, which is stored in a storage medium and implements the risk identification method provided in the above embodiments when the computer program product is executed by a computer.
[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0164] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A risk identification method, characterized in that, include: By using at least one transformation layer of a multimodal large model, first feature information is obtained based on the first text and the first image. The first feature information includes: image features of the first image output by each transformation layer, text features of the first text, and text description features of the first image. Based on the image features and text description features output by each conversion layer, a first conversion layer is determined from the at least one conversion layer, wherein the image features and text description features output by the first conversion layer have the highest similarity. Based on the image features and text description features output by the first conversion layer, a first index of the first conversion layer is obtained, and based on the image features and text features output by the first conversion layer, a second index of the first conversion layer is obtained. The first index is used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index is used to characterize the semantic deviation between the first image and the first text. Based on the first index and the second index, a first determination result is obtained, which is used to characterize whether the first text and the first image have an attack risk.
2. The risk identification method according to claim 1, characterized in that, The step of obtaining first feature information based on the first text and the first image through at least one transformation layer of a multimodal large model includes: Based on the first image, obtain a text description of the first image; The first text, the first image, and the text description of the first image are input into each of the at least one conversion layer to obtain the first feature information.
3. The risk identification method according to claim 1, characterized in that, Determining the first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer includes: Based on the image and text features output by each transformation layer, obtain the first similarity of each transformation layer; Obtain the maximum value of the first similarity of each conversion layer, and determine the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
4. The risk identification method according to claim 1, characterized in that, The step of obtaining the second index of the first conversion layer based on the image features and text features output by the first conversion layer includes: Obtain the covariance similarity between the image features and text features output by the first conversion layer, and obtain the second index based on the covariance similarity.
5. The risk identification method according to any one of claims 1 to 4, characterized in that, The step of obtaining the first determination result based on the first index and the second index includes: Obtain the first difference between the second index and the first index; If the first difference is greater than or equal to the first preset threshold, then the first determination result is that the first text and the first image are at risk. If the first difference is less than the first preset threshold, then the first determination result is that the first text and the first image are risk-free.
6. A risk identification device, characterized in that, include: Get the module; The acquisition module is used to acquire first feature information based on the first text and the first image through at least one conversion layer of the multimodal large model. The first feature information includes: image features of the first image output by each conversion layer, text features of the first text, and text description features of the first image. The acquisition module is further configured to determine a first conversion layer from the at least one conversion layer based on the image features and text description features output by each conversion layer, wherein the image features and text description features output by the first conversion layer have the highest similarity. The acquisition module is further configured to acquire a first index of the first conversion layer based on the image features and text description features output by the first conversion layer, and to acquire a second index of the first conversion layer based on the image features and text features output by the first conversion layer. The first index is used to characterize the semantic deviation between the first image and the text description features of the first image when there is no attack, and the second index is used to characterize the semantic deviation between the first image and the first text. The acquisition module is further configured to acquire a first determination result based on the first index and the second index, wherein the first determination result is used to characterize whether the first text and the first image have an attack risk.
7. The risk identification device according to claim 6, characterized in that, The acquisition module is specifically used for: Based on the first image, obtain a text description of the first image; The first text, the first image, and the text description of the first image are input into each of the at least one conversion layer to obtain the first feature information.
8. The risk identification device according to claim 6, characterized in that, The acquisition module is specifically used for: Based on the image and text features output by each transformation layer, obtain the first similarity of each transformation layer; Obtain the maximum value of the first similarity of each conversion layer, and determine the conversion layer corresponding to the maximum value of the first similarity of each conversion layer as the first conversion layer.
9. The risk identification device according to claim 6, characterized in that, The acquisition module is specifically used for: Obtain the covariance similarity between the image features and text features output by the first conversion layer, and obtain the second index based on the covariance similarity.
10. The risk identification device according to any one of claims 6 to 9, characterized in that, The acquisition module is specifically used for: Obtain the first difference between the second index and the first index; If the first difference is greater than or equal to the first preset threshold, then the first determination result is that the first text and the first image are at risk. If the first difference is less than the first preset threshold, then the first determination result is that the first text and the first image are risk-free.
11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the risk identification method as described in any one of claims 1-5.
12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the risk identification method as described in any one of claims 1-5.
13. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the risk identification method as described in any one of claims 1-5.