Multimodal Large Model Defense Method and Device Based on Multi-Agent Intent Analysis

By employing a multi-agent intent analysis method and leveraging the collaborative work of the visual tower encoder and agents, the fine-grained problem of visual filtering in large multimodal models is solved, thereby improving security and usability of interaction and avoiding issues such as high deployment costs and poor portability.

CN123115359APending Publication Date: 2026-11-17TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611601330.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies lack fine-grained visual filtering methods before multimodal fusion in large multimodal models. User questions can be manipulated by attackers, the filtering process disrupts the distribution of visual features, and the deployment cost is high with poor portability.

Method used

A multi-agent intent analysis method is adopted, which utilizes the visual tower encoder, image understanding agent, phrase cue extraction agent and intent inference agent to work together to generate visual patch relevance score sequence. Through index filtering and spatial order-preserving rearrangement, fine-grained filtering of harmful visual information is achieved, avoiding retraining and fine-tuning.

Benefits of technology

It achieves fine-grained visual information filtering, prevents harmful information from being preferentially retained, maintains the normal capabilities of the model, reduces the risk of attacker manipulation, and improves security and usability of interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115359A_ABST
    Figure CN123115359A_ABST
Patent Text Reader

Abstract

This application provides a multimodal large model defense method and apparatus based on multi-agent intent analysis, which can be applied to the fields of artificial intelligence security and large model content security protection. The method includes: encoding an input image using a visual tower encoder to obtain an original visual patch embedding sequence; performing image understanding, phrase cue extraction, and intent inference on the input image using a multi-agent agent to obtain the multi-agent inferred intent; performing a correlation score on the original visual patch embedding sequence and the multi-agent inferred intent using an auxiliary visual-text alignment sub-model to obtain a visual patch correlation score sequence; performing index filtering and spatial order-preserving rearrangement operations on the original visual patch embedding sequence using the visual patch correlation score sequence to obtain a filtered visual patch embedding sequence; and performing multimodal fusion processing on the filtered visual patch embedding sequence and the original user question using a multimodal large model to generate a secure answer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence security and large model content security protection technology, specifically to the fields of multimodal large model human-computer interaction security detection, multi-agent collaborative intent recognition, large model prompt word injection and harmful input defense technology, and more specifically to a multimodal large model defense method and device based on multi-agent intent analysis. Background Technology

[0002] With the widespread deployment of Multimodal Large Language Models (MLLM), attackers can steganographically embed harmful commands (such as jailbreak commands, malicious attack commands, etc.), illegal text, and suggestive patterns into local areas of input images. They can then use seemingly compliant user questions to induce the model to output harmful content (such as malicious code, illegal or inappropriate speech, etc.). Therefore, for visual input information from MLLM, it is necessary to perform fine-grained filtering and defense on visual regions carrying harmful information before multimodal information fusion.

[0003] Related technical solutions include input-side detection schemes that combine rule matching with independent modality classification, content moderation schemes based on a single security agent, and vision token filtering schemes that use user questions as queries. However, the related technical solutions have the following problems: (1) The processing granularity remains at the whole image level, lacking fine-grained visual filtering methods before multimodal fusion; (2) Using the original user questions as guiding information for visual filtering, the filtering benchmark itself can be manipulated by attackers; (3) The filtering process changes the distribution of original visual features or destroys the image spatial structure, impairing the model's normal text-image question answering ability; (4) It relies on retraining or fine-tuning the model's backbone network, resulting in high deployment costs and poor transferability. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a multimodal large model defense method and apparatus based on multi-agent intent analysis.

[0005] The first aspect of this application provides a multimodal large-scale model defense method based on multi-agent intent analysis, comprising: encoding an input image using a visual tower encoder of the multimodal large-scale model to obtain an original visual patch embedding sequence; performing global semantic parsing and content recognition on the input image using an image understanding agent to obtain multi-dimensional semantic information of the input image; generating a global image description text based on the multi-dimensional semantic information; performing local text identifier detection on the input image using a phrase cue extraction agent to obtain phrase cue; and fusing the global image description text and phrase cue using an intent inference agent to obtain the multi-agent inferred intent of the input image; performing a correlation score between each visual patch in the original visual patch embedding sequence and the multi-agent inferred intent using an auxiliary visual-text alignment sub-model of the multimodal large-scale model to obtain a visual patch correlation score sequence; performing an index filtering operation and a spatial order-preserving rearrangement operation on the original visual patch embedding sequence using the visual patch correlation score sequence to obtain a filtered visual patch embedding sequence; and performing multimodal fusion processing on the filtered visual patch embedding sequence and the original user question using the multimodal large-scale model to generate a safe answer that filters out risky information.

[0006] According to embodiments of this application, the above-mentioned encoding of the input image using a multimodal large-scale visual tower encoder to obtain the original visual patch embedding sequence includes: preprocessing the input image and normalizing the pixel values ​​to obtain a standard input image; dividing the standard input image into multiple non-overlapping image blocks; mapping the pixel vectors in the multiple non-overlapping image blocks to feature vectors with a preset dimension to obtain multiple initial visual patch embeddings; adding positional encoding to each initial visual patch embedding to obtain multiple visual patch embeddings with spatial positional information; performing feature update operations and semantic association operations based on a self-attention interaction mechanism on each visual patch embedding with spatial positional information to obtain multiple updated visual patch embeddings; and normalizing the multiple updated visual patch embeddings to obtain the original visual patch embedding sequence.

[0007] According to embodiments of this application, the above-described method of using an image understanding agent to perform global semantic analysis and content recognition on an input image to obtain multi-dimensional semantic information of the input image, and generating a global image description text based on the multi-dimensional semantic information, includes: performing image size normalization and pixel value standardization on the input image to obtain a preprocessed input image; extracting visual features from the preprocessed input image to obtain image visual features; performing target object detection, scene classification, person detection, human action recognition, and text content recognition operations on the preprocessed input image based on the image visual features to obtain object information, scene information, person information, action information, and image embedded text information; integrating the object information, scene information, person information, action information, and image embedded text information into structured multi-dimensional semantic information, and generating a global image description text representing the global content of the input image based on the multi-dimensional semantic information.

[0008] According to an embodiment of this application, the above-mentioned method of using a phrase cue extraction agent to perform local text identifier detection on an input image to obtain phrase cues includes: filtering local text, local identifiers, and / or local screenshot text information of the input image to obtain local text information filtering results; if the local text information filtering results are valid local text that meets the preset reference value, extracting text features from the local text information filtering results to obtain phrase cues; if the local text information filtering results are valid local text that does not meet the preset reference value, setting the phrase cues to null.

[0009] According to embodiments of this application, the above-mentioned multi-agent inference intent of the input image by fusing global image description text and phrase cues with an intent inference agent includes: when phrase cues are empty, performing semantic reasoning on the global image description text to obtain multi-agent inference intent; when phrase cues are not empty, fusing the global image description text and phrase cues, and performing semantic reasoning on the fusion result to obtain multi-agent inference intent.

[0010] According to embodiments of this application, the above-mentioned auxiliary visual-text alignment sub-model using a multimodal large model performs correlation scoring on each visual patch in the original visual patch embedding sequence and the multi-agent inference intent to obtain a visual patch correlation score sequence, including: encoding the input image using the auxiliary visual encoder of the auxiliary visual-text alignment sub-model to obtain a visual patch-level feature sequence corresponding to the visual patches in the original visual patch embedding sequence; encoding the multi-agent inference intent using the auxiliary text encoder of the auxiliary visual-text alignment sub-model to obtain a text vector; normalizing the visual patch-level feature sequence to obtain a normalized visual patch-level feature sequence; and calculating the similarity between the text vector and each normalized visual patch-level feature in the normalized visual patch-level feature sequence to obtain a visual patch correlation score sequence.

[0011] According to embodiments of this application, the aforementioned auxiliary visual-text alignment sub-model includes a batch comparison-based image-text semantic alignment pre-training sub-model, a pairwise decision-based image-text semantic alignment pre-training sub-model, or a noisy image-text embedding alignment pre-training sub-model; wherein, the similarity includes cosine similarity, Euclidean distance similarity, and inner product similarity.

[0012] According to an embodiment of this application, the above-mentioned index filtering operation and spatial order-preserving rearrangement operation performed on the original visual patch embedding sequence using the visual patch relevance score sequence to obtain the filtered visual patch embedding sequence includes: filtering the visual patch relevance score sequence based on descending index to obtain a candidate index set; sorting the candidate index set in ascending order according to the spatial position of the visual patch embedding in the input image to obtain an order-preserving index set; and filtering the original visual patch embedding sequence according to the order-preserving index set to obtain the filtered visual patch embedding sequence.

[0013] According to embodiments of this application, the above-mentioned multimodal fusion processing of the selected visual patch embedding sequence and the original user question using a multimodal large model to generate a safe answer that filters out risk information includes: using the multimodal projector of the multimodal large model to map the selected visual patch embedding sequence to the language model feature space to obtain the projected visual patch embedding sequence; performing multimodal fusion processing on the projected visual patch embedding sequence and the original user question; and based on the result of the multimodal fusion processing, using the language model autoregression of the multimodal large model to generate a safe answer that filters out risk information.

[0014] A second aspect of this application provides a multimodal large model defense device based on multi-agent intent analysis, comprising: an input acquisition unit for acquiring an input image and an original user question; a visual tower encoding unit for encoding the input image using a visual tower encoder of a multimodal large model to obtain an original visual patch embedding sequence; and a multi-agent intent analysis unit for using an image understanding agent to perform global semantic parsing and content recognition on the input image to obtain multi-dimensional semantic information of the input image, generating a global image description text based on the multi-dimensional semantic information, using a phrase cue extraction agent to perform local text identifier detection on the input image to obtain phrase cues, and using an intent inference agent to combine the global image description text and the phrase cue. The system integrates several key components: a symmetric cues fusion unit to obtain the multi-agent inference intent of the input image; a patch relevance scoring unit to score the relevance between each visual patch in the original visual patch embedding sequence and the multi-agent inference intent using an auxiliary visual-text alignment sub-model of a multimodal large model, resulting in a visual patch relevance score sequence; a visual patch filtering unit to perform index filtering and spatial order-preserving rearrangement operations on the original visual patch embedding sequence using the visual patch relevance score sequence, resulting in a filtered visual patch embedding sequence; and a multimodal projection and response generation unit to perform multimodal fusion processing on the filtered visual patch embedding sequence and the original user question using a multimodal large model, generating a safe response that filters out risky information.

[0015] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0017] A fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] This application provides a multimodal large-scale model defense method based on multi-agent intent analysis. By setting up three types of collaborative agents—image understanding agent, phrase cue extraction agent, and intent inference agent—it uses the ontological information of the input image to generate inferred intent as a filtering guide. This decouples the filtering benchmark from the input text controllable by potential attackers, avoiding interference from leading phrases, jailbreak commands, and format requirements in user questions. It also reduces the risk of attackers preferentially retaining local patches containing steganographic malicious commands through textual inducement, overcoming the vulnerability of the filtering benchmark to manipulation. Using visual patches as the smallest unit of processing, it achieves fine-grained filtering of harmful visual information before multimodal fusion. Simultaneously, the auxiliary visual-text alignment sub-model used in this application neither causes visual feature distribution shifts nor destroys image spatial topological information, maximizing the preservation of the original model's capabilities in normal image-text question-answering tasks such as counting, localization, and layout understanding, overcoming the defect of the filtering process damaging the model's normal capabilities. Furthermore, the method provided in this application avoids the problem of normal requests being blocked entirely due to false positives in binary judgment, ensuring that the output content always aligns with the user's actual needs, improving security while maintaining the model's interactive usability. Attached Figure Description

[0019] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of the multimodal large model defense method based on multi-agent intent analysis according to an embodiment of this application is shown.

[0021] Figure 2 A general flowchart of relevant multimodal large-model defense schemes is shown in the field of multimodal large-model security protection.

[0022] Figure 3 A flowchart of a multimodal large model defense method based on multi-agent intent analysis according to an embodiment of this application is shown.

[0023] Figure 4 A diagram illustrating the architecture of a multimodal large-scale defense mechanism according to an embodiment of this application is shown.

[0024] Figure 5 A flowchart of multi-agent intent analysis according to an embodiment of this application is shown.

[0025] Figure 6 A flowchart illustrating visual patch relevance scoring and order-preserving screening according to an embodiment of this application is shown.

[0026] Figure 7 A structural diagram of a multimodal large-scale defense device based on multi-agent intent analysis according to an embodiment of this application is shown.

[0027] Figure 8 A block diagram of an electronic device suitable for implementing a multimodal large model defense method based on multi-agent intent analysis, according to an embodiment of this application, is shown. Detailed Implementation

[0028] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0033] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided in this application all provide users with corresponding operation entry points for users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process enters the expert decision-making process.

[0034] As used herein, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, thereby generating a corresponding output for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this application, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably throughout this application.

[0035] It's important to note that the term "neural network" can refer to a machine learning network based on deep learning. A neural network processes input and provides corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between them. Neural networks used in deep learning applications often include many hidden layers, increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer serves as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output becomes the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each processing the input from the layer above.

[0036] It should be understood that machine learning generally includes three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0037] In one or more embodiments described in this application, the term "large model" can refer to a deep learning model with a large number of model parameters, which can include hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model / basic model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such a model can adapt to a wide range of downstream tasks and has good generalization ability, such as large language models and multimodal pre-trained models. It should be understood that in practical applications, large models only require a small number of samples to fine-tune the pre-trained model before it can be applied to different tasks. Large models can be widely used in natural language processing, computer vision, and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering, image description, and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. Major application scenarios for large models can include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0038] Figure 1 An application scenario diagram of the multimodal large model defense method based on multi-agent intent analysis according to an embodiment of this application is shown.

[0039] like Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0041] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0042] It should be noted that the multimodal large model defense method based on multi-agent intent analysis provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the multimodal large model defense device based on multi-agent intent analysis provided in this application embodiment can generally be set in server 105 and / or terminal devices 101-103.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] In the field of multimodal large model security protection, how to overcome the problems of the lack of visual input filtering means in the MLLM inference link, the susceptibility to inducement and manipulation of speech when using user questions as visual screening guides, the destruction of the original visual feature distribution and spatial structure in the filtering process, and the difficulty in low-cost deployment due to reliance on retraining, and provide a defense technology solution that performs fine-grained filtering of visual regions carrying harmful information before multimodal information fusion has become the research focus of the field of multimodal large model security protection.

[0045] In the field of multimodal large-scale model security protection, there are currently three main types of related technical solutions for harmful visual inputs.

[0046] The first category is an input-side detection scheme that combines rule matching with independent modality classification. This scheme constructs basic interception rules by manually configuring sensitive words and regular expressions, and then trains text classification models and image classification models separately. It performs independent feature extraction and risk scoring on the input text and images, and uses a simple weighted method to fuse the judgment results of the two modalities, outputting a binary conclusion of safe or harmful, and then performs overall permission or overall interception accordingly.

[0047] The second type is a content moderation solution based on a single security agent. This solution deploys a security agent to receive multimodal input, uses a fixed prompt template to call a basic large model to perform a one-time risk assessment of the text and image content, and directly outputs a pass or block instruction. This solution has the same processing granularity as the first type, using the entire image as the smallest processing unit.

[0048] The third category is a visual lexical filtering scheme based on user questions as queries, which is the closest prior art to this application. After the visual tower completes the encoding, this scheme uses the original question input by the user as the query text, calculates the relevance between each visual patch and the question text, and retains the patches with higher relevance to the downstream. Its original design goal is to compress the number of visual lexical units and reduce inference overhead.

[0049] Figure 2 A general flowchart of relevant multimodal large-model defense schemes is shown in the field of multimodal large-model security protection.

[0050] like Figure 2 As shown, the general process of the relevant multimodal large model defense method is as follows: user submits multimodal input → independent detection of text branch and image branch → simple weighted fusion of the two modal detection results → binary risk judgment → execution of release or interception.

[0051] However, the relevant technical solutions have the following problems: (1) The processing granularity is limited to the whole image level, and there is a lack of fine-grained visual filtering methods before multimodal fusion. Both the first and second types of solutions take the whole image as the smallest processing unit. Once it is determined to be safe, all visual patches of the image will be completely sent into the multimodal projector and language model. When harmful information is only hidden in a local area of ​​the image, and the image as a whole presents a normal scene, the risk score of the whole image level is diluted by the features of a large number of normal areas, making it difficult to trigger the interception threshold. The hidden harmful area then enters the language model context along with the normal area. The technical reason is that the existing solutions deploy the defense link before the model input end. The target is the original image file rather than the patch embedding sequence output by the visual tower. It does not have the ability to remove local patches at the representation level. (2) The original user question is used as the guiding information for visual screening. The screening benchmark itself can be manipulated by attackers. The third type of scheme calculates the correlation between the patch and the user question and retains the high-scoring patch. This mechanism is effective in normal scenarios, but has structural defects in adversarial scenarios: the user question is completely controllable by the attacker. The attacker can construct inducing words that point to the steganographic area, such as asking the model to read and execute the text in the image, so that the patch with the steganographic harmful instruction gets the highest relevance score and is retained first, while the normal scenario patch is eliminated. The technical reason is that the design goal of this scheme is to accelerate inference rather than to protect security. Its query text and the attack surface being defended come from the same untrusted source, and the filtering benchmark and the attacker's input are not isolated. (3) The filtering process changes the original visual feature distribution or destroys the image spatial structure, which impairs the model's normal image and text question answering ability. Some schemes directly use the visual features output by the auxiliary alignment model as downstream input after filtering, or reorder the retained patches according to the relevance score and send them to the projector. The former makes the feature distribution obtained downstream inconsistent with the distribution during the training of the visual tower, and the latter makes the position order of the patch sequence no longer correspond to the position in the two-dimensional space of the image. The technical reason is that the existing solutions do not distinguish between the scoring index decision and the downstream feature source, nor do they recognize that the order of the patch sequence carries the spatial topological information of the image, which leads to a decrease in the accuracy of the model in normal tasks such as counting, localization, and layout understanding. (4) It relies on retraining or fine-tuning the backbone network of the model, which has high deployment costs and poor transferability. Some security enhancement solutions require fine-tuning the language model or projector using security alignment data, or inserting adaptation modules that need to be trained into the model. The technical reason is that the defense logic is coupled with the model parameters. Each time the basic model is changed, it needs to be retrained. Moreover, fine-tuning will cause the model to regress in performance on general tasks, making it difficult to deploy in a low-intrusive manner on the already launched inference service.

[0052] To address at least one of the aforementioned technical problems, this application provides a multimodal large model defense method based on multi-agent intent analysis. The following detailed description of the multimodal large model defense method based on multi-agent intent analysis provided by this application is provided through embodiments, specific implementation methods, and in conjunction with the accompanying drawings.

[0053] Figure 3 A flowchart of a multimodal large model defense method based on multi-agent intent analysis according to an embodiment of this application is shown.

[0054] like Figure 3 As shown, the multimodal large model defense method 300 based on multi-agent intent analysis according to an embodiment of this application may include steps S310 to S350.

[0055] In step S310, the input image is encoded using a multimodal large model visual tower encoder to obtain the original visual patch embedding sequence.

[0056] Multimodal large models, or multimodal large language models, refer to a class of large-scale pre-trained models that can simultaneously receive, understand, and process information from two or more different modalities; modalities include images, text, audio, etc.

[0057] Vision Tower is a built-in visual coding module for multimodal large models, primarily implementing functions such as image preprocessing, image feature extraction, and dimension mapping adaptation. The Vision Tower used in this application includes an image preprocessing submodule, an image segmentation and patch embedding submodule, a positional coding submodule, a visual feature coding submodule, and a normalized output submodule.

[0058] The image preprocessing submodule is used to process the input image. Perform resizing, pixel value normalization, and channel normalization to obtain a standard input image that meets the Vision Tower input specifications. The image tiling and patch embedding submodule is used to tile the standard input image according to a preset image patch size. Divided into Each of the following non-overlapping image patches is used, and the pixel data of each image patch is expanded or mapped to an initial visual patch of a preset dimension for embedding. For the first... The initial visual patch embedding of an image patch can be represented as: .in, Indicates the input image from standard input. The first obtained Image blocks, This represents the mapping function used to map image patches to the visual feature space. This indicates the corresponding bias parameter.

[0059] The positional encoding submodule is used to add positional encoding to the corresponding initial visual patch embedding based on the spatial position of each image patch in the input image. A visual patch embedding with spatial location information can be represented as: ,in, Indicates the first The positional encoding corresponds to each image patch. By adding positional encoding, the subsequent visual feature encoding process can distinguish the relative or absolute spatial positions of different visual patches in the original image.

[0060] The visual feature encoding submodule can employ a multi-layer visual encoding network based on a self-attention mechanism. In one optional implementation, the visual feature encoding submodule includes multiple visual Transformer encoding layers connected in sequence, each visual Transformer encoding layer including a multi-head self-attention unit and a feedforward network unit. For the... The feature update process of the layered visual Transformer encoding layer can be represented as follows: ; ,in, Indicates entering the first The visual patch feature sequence before the layer, This indicates that the visual patch feature sequence is obtained after layer normalization, multi-head self-attention calculation, and the first residual connection. Layer intermediate feature representation; The intermediate feature represents the first value obtained after further layer normalization, feedforward network calculation, and a second residual connection. Layer output feature representation, superscript Indicates the layer number of the visual Transformer coding layer, symbol Feature representations used to distinguish different processing stages within the same visual Transformer encoding layer do not represent mathematical differentiation operations. LN represents normalization, MSA represents multi-head self-attention, and FFN represents feedforward network operation. Through self-attention interactions between multiple visual patches, each visual patch retains its own local visual information while fusing contextual information from other visual patches, thereby obtaining a visual representation with global semantic association capabilities.

[0061] The normalization output submodule is used to normalize the features of each visual patch output from the last layer of the visual coding network and form the original visual patch embedding sequence: ,in, Indicates the first An original visual patch is embedded. The range of values ​​is , This indicates the number of visual patches corresponding to the input image. (A positive integer greater than or equal to 2) This indicates the dimension in which the visual patch is embedded.

[0062] It should be noted that the aforementioned visual tower structure can reuse the existing visual encoding network of the original multimodal large model, and its model parameters remain frozen during the defense process provided in this application. This application does not change the internal parameters of the visual tower, but performs subsequent relevance scoring and filtering on the visual patches after the visual tower obtains the original visual patch embedding sequence and before embedding the visual patches into the input multimodal projector.

[0063] In step S320, the image understanding agent performs global semantic parsing and content recognition on the input image to obtain multi-dimensional semantic information of the input image. Based on the multi-dimensional semantic information, a global image description text is generated. The phrase cue extraction agent performs local text identifier detection on the input image to obtain phrase cues. Finally, the intent inference agent fuses the global image description text and phrase cues to obtain the multi-agent inferred intent of the input image.

[0064] In one optional implementation, in order to formally describe the generation of output information from different agents and the subsequent fusion process, the information obtained by different agents can first be semantically represented, and after obtaining the corresponding semantic information, the intention inference agent can fuse the output information of different agents to obtain a multi-agent inference intention that represents the potential intention of the input image.

[0065] Specifically, the image understanding agent processes the input image. The input-output relationship of global semantic parsing can be represented as follows: .

[0066] in, Indicates the input image. This represents the image semantic parsing function executed by the image understanding agent. This indicates that the image understanding agent understands the input image. The obtained global image semantic representation mapping relationship. This global image semantic mapping relationship is used to characterize the main objects, scene content, object attributes, and semantic relationships between objects in the input image, and serves as the semantic basis for generating global image description text and subsequent multi-agent intent inference. It should be noted that... This describes the mapping relationship between the global image semantic information obtained by the image understanding agent from the input image, and does not represent the weighted fusion process between the output information of different agents. Weighted fusion is further executed by the intent inference agent after obtaining the global image description information output by the image understanding agent and the phrase cues output by the phrase cue extraction agent, so as to comprehensively utilize the global semantic information and local textual semantic information of the input image to obtain the multi-agent inference intent for subsequent visual patch relevance evaluation.

[0067] Step S320 above involves multi-agent collaborative intent analysis to generate inferred intent; the processing object is the input image. The input image is synchronously sent to the multi-agent intent analysis unit, which contains three types of collaborative agents: image understanding agent, phrase cue extraction agent, and intent inference agent.

[0068] In step S330, the auxiliary visual-text alignment sub-model of the multimodal large model is used to score the relevance between each visual patch in the original visual patch embedding sequence and the multi-agent inference intent, so as to obtain the visual patch relevance score sequence.

[0069] Step S330 above involves visual patch relevance scoring. The processing objects are the input image and the inferred intent. An auxiliary visual-text alignment sub-model is loaded, and the auxiliary visual-text alignment sub-model is used to extract visual patch block-level features and inferred intent text features, respectively.

[0070] In step S340, the original visual patch embedding sequence is subjected to index filtering and spatial order-preserving rearrangement operations using the visual patch relevance score sequence to obtain the filtered visual patch embedding sequence.

[0071] In step S350, the selected visual patch embedding sequence and the original user question are fused using a multimodal large model to generate a safe answer that filters out risk information.

[0072] The risk information includes jailbreak commands, prohibited content, illegal content, malicious attack commands, privacy-stealing commands, deceptive content, and antagonistic remarks.

[0073] Regarding the original user issue, in the embodiments of this application, user consent or authorization can be obtained before acquiring user information. Simultaneously, in the embodiments of this application, a corresponding operation entry point can be provided for the user to choose to agree to or reject the automated decision-making result. That is, before processing / making a decision on the user information, the user's instruction to agree to or reject the processing / decision can be obtained through the corresponding operation entry point. If the user agrees to the processing / decision, the user information is processed / decided, i.e., the relevant steps are executed. If the user refuses to perform the processing / decision, the expert decision-making process is initiated.

[0074] This application provides a multimodal large-scale model defense method based on multi-agent intent analysis. By setting up three types of collaborative agents—image understanding agent, phrase cue extraction agent, and intent inference agent—it uses the ontological information of the input image to generate inferred intent as a filtering guide. This decouples the filtering benchmark from the input text controllable by potential attackers, avoiding interference from leading phrases, jailbreak commands, and format requirements in user questions. It also reduces the risk of attackers preferentially retaining local patches containing steganographic harmful instructions through textual inducement, overcoming the vulnerability of the filtering benchmark to manipulation. Using visual patches as the smallest unit of processing, it achieves fine-grained filtering of harmful visual information before multimodal fusion. Furthermore, the auxiliary visual-text alignment sub-model used in this application reduces the disturbance of the original visual tower's output feature distribution to auxiliary feature processing without destroying the image spatial topology information, maximizing the preservation of the original model's capabilities in normal image-text question-answering tasks such as counting, localization, and layout understanding, overcoming the defect of the filtering process damaging the model's normal capabilities. Furthermore, the method provided in this application can avoid the problem of normal requests being blocked as a whole due to false positives in binary decision-making, ensuring that the output content always matches the user's real needs, and maintaining the interactive usability of the model while improving security.

[0075] In one specific implementation, the multimodal large model includes a visual tower, a multimodal projector, and a language model, with an auxiliary visual-text alignment sub-model to achieve image-text feature encoding. Compared with related prior art, the technical solution of this application includes: First, constructing a multi-agent collaborative intent analysis mechanism to generate inferred intent based on the image's own information; Second, decoupling the intent analysis path from the answer generation path, with the inferred intent only used for visual patch selection and not participating in the final answer generation, while the original user question is fully preserved; Third, only using the output features of the auxiliary alignment model to calculate relevance and determine the patch index, with the selected features taken from the original visual patch embedding of the visual tower; Fourth, rearranging the patch index after selection according to the original image space order, preserving the image space structure information; Fifth, embedding the entire process between the visual tower and the multimodal projector, freezing all backbone parameters of the original multimodal large model throughout, eliminating the need for retraining and fine-tuning.

[0076] Figure 4 A diagram illustrating the architecture of a multimodal large-scale defense mechanism according to an embodiment of this application is shown.

[0077] like Figure 4 As shown, the input image Enter the intent analysis path and the answer generation path respectively.

[0078] In the intent analysis path, the image understanding agent and the phrase cue extraction agent respectively generate global image description text. and optional phrase clues The intention is to infer the intelligent agent's description of text based on the global image. and phrase clues Generate inference intent The guide text embedding is obtained by encoding with an auxiliary text encoder. .

[0079] In the response generation path, the auxiliary visual encoder extracts patch-level features from the input image and combines them with guided text embeddings. Calculate the relevance score for each visual patch. ,…, ,…, ,in, Indicates the first Each correlation score, The range of values ​​is ( Candidate index sets are obtained by Top-K selection (where the integers are positive integers greater than or equal to 2). The visual patches are then sorted in order-preserving order according to their spatial location in the original image, resulting in an ordered index set. .

[0080] On the other hand, the original multimodal large model visual tower encodes the input image to obtain the original visual patch embedding sequence. According to the order-preserving index set The filtered original visual patch embedding sequence is extracted from the original visual patch embedding sequence. . After mapping by the multimodal projector of the original multimodal large model, it corresponds to the original user problem without modification. The input is shared into a large language model to generate the final answer. .

[0081] The features output by the auxiliary visual encoder are used only for relevance calculation and patch index determination, and are not directly input into the multimodal projector; inference intent. It only participates in visual patch selection and does not participate in final answer generation; the visual tower, multimodal projector, and large language model parameters of the original multimodal large model remain frozen throughout the process.

[0082] According to embodiments of this application, the above-mentioned encoding of the input image using a multimodal large-scale visual tower encoder to obtain the original visual patch embedding sequence includes: preprocessing the input image and normalizing the pixel values ​​to obtain a standard input image; dividing the standard input image into multiple non-overlapping image blocks; mapping the pixel vectors in the multiple non-overlapping image blocks to feature vectors with a preset dimension to obtain multiple initial visual patch embeddings; adding positional encoding to each initial visual patch embedding to obtain multiple visual patch embeddings with spatial positional information; performing feature update operations and semantic association operations based on a self-attention interaction mechanism on each visual patch embedding with spatial positional information to obtain multiple updated visual patch embeddings; and normalizing the multiple updated visual patch embeddings to obtain the original visual patch embedding sequence.

[0083] The above embodiments involve input acquisition and raw visual patch encoding. In one specific implementation, the processing object is an input image. Original user issues The input acquisition unit receives the input image submitted by the user. Original user issues The input image is forwarded to the visual tower coding unit. The visual tower coding unit encodes the image using the original MLLM's built-in visual tower, outputting the original visual patch embedding sequence: .in Represents the original visual patch embedding sequence. Represents the visual tower output number Embed an original visual patch ( The range of values ​​is ), The total number of visual patches obtained from the image segmentation is represented by the total number of visual patches. , This represents the visual patch embedding dimension. Visual tower encoding is a common technical feature; the distinguishing feature of this application is that the original visual patch embedding is not directly fed into the multimodal projector but requires subsequent filtering. The beneficial effect of this technical feature is that it reuses the original visual encoder to extract native visual features without modifying the backbone network, ensuring that the basic visual representation does not shift.

[0084] According to embodiments of this application, the above-described method of using an image understanding agent to perform global semantic analysis and content recognition on an input image to obtain multi-dimensional semantic information of the input image, and generating a global image description text based on the multi-dimensional semantic information, includes: performing image size normalization and pixel value standardization on the input image to obtain a preprocessed input image; extracting visual features from the preprocessed input image to obtain image visual features; performing target object detection, scene classification, person detection, human action recognition, and text content recognition operations on the preprocessed input image based on the image visual features to obtain object information, scene information, person information, action information, and image embedded text information; integrating the object information, scene information, person information, action information, and image embedded text information into structured multi-dimensional semantic information, and generating a global image description text representing the global content of the input image based on the multi-dimensional semantic information.

[0085] In one specific implementation, the image understanding agent performs visual semantic abstraction on the input image, identifies objects, scenes, people, actions, and text content within the image, and generates image description text. (i.e., global image description text).

[0086] Specifically, the image understanding agent includes an image preprocessing submodule, a visual feature extraction submodule, a multidimensional visual semantic parsing submodule, a semantic information structuring submodule, and an image description generation submodule. The multidimensional visual semantic parsing submodule further includes target object recognition branches, scene recognition branches, person recognition branches, human action recognition branches, and image embedded text recognition branches.

[0087] The image understanding agent receives the input image. Then, the image preprocessing submodule is first used to process the input image. Perform size normalization, pixel value normalization, and channel format conversion to obtain the preprocessed image. Then, the preprocessed image The input visual feature extraction submodule extracts visual features from the image through a visual encoding network. The processing procedure can be represented as follows: ; ,in, This represents the image preprocessing function of an image understanding agent. This represents the visual feature extraction function of an image understanding agent.

[0088] In obtaining image visual features Subsequently, the multidimensional visual semantic parsing submodule is based on Semantic analysis is performed on different dimensions. Specifically, the target object recognition branch identifies the main objects and their categories in the input image, and further determines the location regions of these objects; the scene recognition branch determines the scene category and attributes corresponding to the input image; the person recognition branch determines the presence of people in the image and their location; the human motion recognition branch determines human motion information based on the person's region and the relationship between the person and surrounding objects; and the embedded text recognition branch identifies textual information in the input image that has global descriptive value.

[0089] The above process can be abstractly represented as: ; ; ; ; ,in, Represents object information, Indicates scene information, Indicates information about a person. Indicates action information, This indicates text information embedded in the image; , , , and These represent the corresponding semantic parsing functions.

[0090] The semantic information structuring submodule organizes and merges the information output from the different branches according to preset semantic fields, forming structured multi-dimensional semantic information. For example, structured multi-dimensional semantic information. It can include at least scene fields, core object fields, character fields, action fields, object relationship fields, and image / text fields, which can be represented as follows: ,in, A structured fusion function representing multi-dimensional semantic information.

[0091] Furthermore, the image description generation submodule, based on structured multi-dimensional semantic information... Semantic organization is performed to convert discrete object information, scene information, character information, action information, and embedded text information into a natural language description that can represent the overall content of the input image, resulting in image description text. : ,in, This represents the image description generation function.

[0092] For example, if structured multi-dimensional semantic information indicates that the image scene is an office environment, the main objects include a computer and a monitor, a person is operating the computer, and there is text content on the monitor, the image description generation submodule can generate image description text according to the semantic order of "scene—core object—person / action—object relationship—recognizable text". The resulting image description text... Instead of simply listing target categories, it provides a unified description of the overall scene, main objects, and their semantic relationships in the input image, providing a global semantic basis for the subsequent intent inference agent.

[0093] According to an embodiment of this application, the above-mentioned method of using a phrase cue extraction agent to perform local text identifier detection on an input image to obtain phrase cues includes: filtering local text, local identifiers, and / or local screenshot text information of the input image to obtain local text information filtering results; if the local text information filtering results are valid local text that meets the preset reference value, extracting text features from the local text information filtering results to obtain phrase cues; if the local text information filtering results are valid local text that does not meet the preset reference value, setting the phrase cues to null.

[0094] In one specific implementation, the phrase cue extraction agent identifies local text, logos, and screenshot text within an image; if valid local text is identified, phrase cues are output. When there is no local text, phrase clues are not output.

[0095] Specifically, the phrase cue extraction agent may include a local text candidate region detection submodule, a local region preprocessing submodule, a local text recognition submodule, a validity determination submodule, and a phrase semantic induction submodule.

[0096] The local text candidate region detection submodule is used to process the input image. Perform local region scanning to identify candidate regions that may contain text, logos, interface labels, or screenshot text, resulting in a set of candidate local regions: ,in, This represents a local text candidate region detection function. Indicates the first A number of candidate local regions This indicates the number of candidate local regions detected.

[0097] The local region preprocessing submodule performs at least one of the following processing steps on each candidate local region: cropping, scaling, orientation correction, and image enhancement, to reduce the impact of small local text size, tilted orientation, or background interference on the text recognition results. Subsequently, the local text recognition submodule performs text recognition on each processed candidate local region, resulting in a local text candidate set. ,in, Indicates from the first The text content identified in each candidate local region.

[0098] The validity determination submodule filters candidate local texts, removing those with insufficient recognition confidence, containing only meaningless characters, excessively incomplete content, or lacking reference value for subsequent semantic analysis. Validity determination can be based on one or more indicators among text recognition confidence, character completeness, text length, and semantic completeness. For the first... For each local text candidate, its reference value evaluation result can be calculated. And determine whether the candidate local text is a valid local text based on preset judgment conditions: When the effective local text set When empty, it indicates that no local text or phrase clues with pre-defined reference value were obtained. To leave empty, that is: , This represents the empty set.

[0099] When the effective local text set When not empty, the phrase semantic induction submodule performs deduplication, semantic merging, and key information extraction on one or more valid local texts, summarizing lengthy or repetitive local texts into phrase clues that can represent the core meaning of the local text. It can be represented as: ,in, This represents a phrase semantic induction function.

[0100] Therefore, the phrase cue extraction agent does not directly send all the Optical Character Recognition (OCR) text in the input image to the intent inference agent. Instead, it first performs candidate detection, validity screening, and semantic induction on local text, thereby reducing the interference of meaningless characters, background text, and low-confidence recognition results on the subsequent intent inference process.

[0101] According to embodiments of this application, the above-mentioned multi-agent inference intent of the input image by fusing global image description text and phrase cues with an intent inference agent includes: when phrase cues are empty, performing semantic reasoning on the global image description text to obtain multi-agent inference intent; when phrase cues are not empty, fusing the global image description text and phrase cues, and performing semantic reasoning on the fusion result to obtain multi-agent inference intent.

[0102] In one specific implementation, the intention is to infer the fusion of image description text by the intelligent agent. Optional phrase clues Abstracting to obtain the inference intent that represents the core semantics of the image The calculation method is as follows: when phrase clues exist: When there are no phrase clues: .in To infer the agent's reasoning function. Describe text for images. Using phrase clues, To infer intent.

[0103] Specifically, the intent inference agent may include an input organization submodule, a semantic encoding submodule, a semantic fusion submodule, an intent reasoning submodule, and an intent generation submodule.

[0104] The input organization submodule receives the image description text output by the image understanding agent. And determine whether the phrase cue extraction agent outputs a valid phrase cue. It should be noted that the input to the agent intending to infer the intelligence does not include the original user question. This isolates the intent analysis path from the user-controllable original problem at the data input level.

[0105] The semantic encoding submodules respectively process the image description text Non-empty phrase clues Semantic encoding is performed to obtain a global image semantic representation. and local text semantic representation : ; ,in, This indicates the text semantic encoding function used by the agent to infer intent.

[0106] When phrase clues When empty, the semantic fusion submodule will represent the global image semantics. Directly used as the semantic representation to be inferred ,Right now: .

[0107] When phrase clues When not empty, the semantic fusion submodule represents the global image semantics. and local text semantic representation Semantic fusion is performed so that the overall scene semantics and local textual semantics of the image simultaneously participate in intent determination. This process can employ feature concatenation, attention fusion, or context fusion based on a semantic reasoning model, and can be uniformly represented as: ,in, This represents a semantic fusion function.

[0108] In one alternative implementation, when the global image semantic representation... and local text semantic representation When features have the same dimension, the semantic fusion function can employ an adaptive fusion method based on attention weights, dynamically determining the contribution of global image semantic information and local text semantic information to the intent inference process based on the semantic content of the current input image. First, the semantic importance scores corresponding to the global image semantic representation and the local text semantic representation are calculated respectively: , ;in, This represents the semantic importance score corresponding to the global image semantic representation. The semantic importance score represents the semantic representation of a local text. and These represent the feature transformation parameters corresponding to the global image semantic representation and the local text semantic representation, respectively. and This represents the corresponding bias parameter. This represents the parameter vector used to map the transformed semantic features to scalar importance scores.

[0109] Furthermore, the semantic importance scores are normalized to obtain the fusion weights corresponding to global image semantic information and local text semantic information: , ;in, The fusion weights represent the semantic information of the global image. The fusion weights represent the semantic information of local texts and satisfy the following conditions: Based on the fusion weights, the global image semantic representation and the local text semantic representation are weighted and fused to obtain the semantic representation to be inferred: Therefore, when phrase cues are not empty, the semantic fusion submodule can adaptively adjust the contribution ratio of global image semantic information and local text semantic information in the fusion process based on the current semantic content of the input image. When the local text has a strong semantic association with the main object or main scene of the input image, the contribution of the local text semantic information in the fusion result can be increased accordingly; when the local text has a low degree of association with the semantics of the main object of the input image, the contribution of the global image semantic information can be increased accordingly, thereby reducing the influence of irrelevant local text, background text, or suggestive text on the subsequent intent inference process.

[0110] It should be noted that the above-mentioned attention-weighted fusion method is a semantic fusion function. This is one possible specific implementation. In other implementations, feature concatenation, cross-attention, context joint encoding, or other methods that can achieve the fusion of global image semantic representation and local text semantic representation can also be used, and this application does not limit these methods.

[0111] In obtaining the semantic representation to be reasoned Subsequently, the intent reasoning submodule is based on the semantic representation to be reasoned. The semantic relationships between visual elements in the input image are analyzed, and the core semantics expressed by the input image are extracted from the combination relationships between objects, scenes, character actions, and local text. The intent generation submodule further organizes the above core semantics into inferred intents for subsequent visual patch selection. ,Right now: .

[0112] It should be noted that the aforementioned global image semantic representation Local text semantic representation and semantic representation to be reasoned Semantic encoding, semantic fusion, and intent generation processes are used to characterize the internal processes of intent inference within an intelligent body; among which, image description text and phrase clues This is to infer the external input information received by the intelligent agent. Specifically, image description text. Non-empty phrase clues After semantic encoding, global image semantic representations are obtained respectively. and local text semantic representation And based on phrase clues Whether the condition is empty, the corresponding semantic fusion and intent generation processes are executed. To summarize the above internal processing from the perspective of the overall input-output relationship of the intent inference agent, the semantic encoding, semantic fusion, and intent generation processes can be uniformly encapsulated into an inference function of the intent inference agent, and the text can be directly described using images. and phrase clues This represents the input to the inference function. Therefore, the above is based on... and The overall representation and based on , and The internal processing of each corresponds to a different processing level, and both have consistent input and output logic.

[0113] Therefore, the above process of inferring intent can also be represented as a whole as follows: ; .in, This represents the semantic reasoning and intent generation function corresponding to the agent in intent inference.

[0114] The generated inference intent Ideally, the inferred intent should be a textual expression that summarizes the core semantics of the input image, rather than an answer to a user question. For example, the inferred intent could focus on "what is the main scene of the input image, what behavior the core object is performing, and what semantic relationship exists between the local text and the main scene." Thus, the inferred intent... It can serve as guiding text when evaluating visual patch relevance in auxiliary vision-text alignment models, while avoiding attacker-controlled issues with the original user. Introduce visual patch selection criteria.

[0115] Inference Intent After generation, it is only sent to the patch relevance scoring unit, where the auxiliary text encoder converts it into a guide text embedding; the intent is then inferred. The language model is not transferred to the original multimodal large model, nor is the original user question replaced or modified. .

[0116] This approach employs a multi-agent collaborative model, generating inferred intent based on image ontology information, instead of directly using original user questions to guide patch selection. The benefits include avoiding interference from leading questions, jailbreak commands, and formatting requirements within the original user questions, and relying on the true semantics of the image for selection, reducing the risk of attackers using text-based manipulation to select harmful visual areas. (Inferred Intent) It is only transmitted to the downstream patch relevance scoring unit and is not sent to the language model to participate in answer generation. The final answer generation always uses the original user question. This creates a decoupled architecture between the intent analysis path and the answer generation path, allowing defense interventions to adjust only the input visual evidence without altering the user's original question, thus ensuring that the output content aligns with the user's true needs.

[0117] The following describes specific implementation methods in conjunction with appendices. Figure 5 The workflow of the multi-agent system provided in this application is described in further detail.

[0118] Figure 5 A flowchart of multi-agent intent analysis according to an embodiment of this application is shown.

[0119] like Figure 5 As shown, the multi-agent intent analysis unit receives the input image. Then, use the same input image The signals are sent to the image understanding agent and the phrase cue extraction agent respectively, forming an independent global image semantic analysis branch and a local text cue analysis branch.

[0120] In the global image semantic analysis branch, the image understanding agent first processes the input image... Image size normalization and pixel standardization are performed to obtain a preprocessed image. Subsequently, a visual feature extraction submodule is used to obtain image visual features, and based on these features, target object recognition, scene recognition, person recognition, human action recognition, and image-embedded text recognition are performed. The image understanding agent structurally combines the object information, scene information, person information, action information, and image-embedded text information output from different semantic analysis branches according to preset fields, and analyzes the semantic relationships between different objects and between objects and people, actions, and scenes, ultimately generating image description text representing the overall content of the input image. .

[0121] In the local text cue analysis branch, the phrase cue extraction agent first detects the input image. The system detects local text regions, local identifier regions, and screenshot text regions. It then performs cropping and text recognition on the detected candidate regions to obtain one or more local text candidates. Subsequently, the phrase cue extraction agent filters these local text candidates based on preset reference value conditions. When no local text meets the preset reference value conditions, the phrase cue is removed. Set to empty; when local text that meets the preset reference value conditions exists, perform semantic induction and key information extraction on the effective local text to generate phrase clues. .

[0122] After completing the above processing, the image understanding agent and the phrase cue extraction agent will extract the image description text c and phrase cues. The signals are sent to the intent inference agent. The intent inference agent first determines the phrase clues. Is it empty? (When phrase clue) When empty, the agent intends to infer based solely on the image description text. Perform semantic reasoning; when phrase clues If not empty, add image description text. The global semantics and phrase cues represented The represented local text semantics are fused, and intent reasoning is performed based on the fused semantic information.

[0123] During the intent reasoning process, the intent inference agent first identifies the core objects and main scenes of the input image, then determines the character's actions, the relationships between objects, and the semantic relationships between local text and their respective visual regions. Based on this, it generates an inferred intent that can summarize the core semantics of the input image. Thus, a collaborative processing relationship is formed among the three agents: "global visual semantic extraction + local text semantic extraction + global / local semantic fusion reasoning," rather than a single agent making a one-time judgment on the input image.

[0124] Furthermore, in the aforementioned multi-agent collaborative process, the three agents can transmit processing results through a pre-defined intermediate data interface. The output interface of the image understanding agent includes at least image description text. The output interface of the phrase cue extraction agent includes phrase cues. Or it could represent a null value indicating the absence of valid phrase clues; the output interface of the intent inference agent includes inferring intent. Original user issues Without undergoing any of the aforementioned semantic fusion processes, and without generating inference intent as an intent inference agent, Input.

[0125] Inference Intent After generation, the multi-agent intent analysis unit will infer the intent. The data is sent to the patch relevance scoring unit, where the auxiliary text encoder of the auxiliary vision-text alignment sub-model infers the intent. It is converted into a text vector, and its correlation with the visual patch-level features of the input image is calculated. Therefore, Figure 5 The final output of the multi-agent collaborative processing shown is not a safe / harmful binary judgment result, but an image-derived inference intent for subsequent visual patch-level screening, thereby decoupling the intent analysis process from the final answer generation process.

[0126] Through the above processing, the image understanding agent is responsible for acquiring the global semantic information of the input image, the phrase cue extraction agent is responsible for extracting the local textual semantic information in the input image, and the intent inference agent further fuses the global semantic information and the optional local textual semantic information to form an inferred intent that represents the core semantics of the input image. Inference of intent The output is sent to the visual patch relevance scoring unit and used as guiding information for subsequent visual patch relevance calculations.

[0127] It should be noted that the stated inference intent This is only used for subsequent visual patch relevance scoring and visual patch selection, and is not used as input for the final answer generation stage; the final answer generation still uses the original user question without modification. This decouples the intent analysis path from the answer generation path, ensuring that the guidance information for visual patch filtering originates from the semantics of the input image itself, rather than from the user-controllable original question.

[0128] According to embodiments of this application, the above-mentioned auxiliary visual-text alignment sub-model using a multimodal large model performs correlation scoring on each visual patch in the original visual patch embedding sequence and the multi-agent inference intent to obtain a visual patch correlation score sequence, including: encoding the input image using the auxiliary visual encoder of the auxiliary visual-text alignment sub-model to obtain a visual patch-level feature sequence corresponding to the visual patches in the original visual patch embedding sequence; encoding the multi-agent inference intent using the auxiliary text encoder of the auxiliary visual-text alignment sub-model to obtain a text vector; normalizing the visual patch-level feature sequence to obtain a normalized visual patch-level feature sequence; and calculating the similarity between the text vector and each normalized visual patch-level feature in the normalized visual patch-level feature sequence to obtain a visual patch correlation score sequence.

[0129] According to embodiments of this application, the aforementioned auxiliary visual-text alignment sub-model includes a batch comparison-based image-text semantic alignment pre-training sub-model, a pairwise decision-based image-text semantic alignment pre-training sub-model, or a noisy image-text embedding alignment pre-training sub-model; wherein, the similarity includes cosine similarity, Euclidean distance similarity, and inner product similarity.

[0130] The above embodiments involve a visual patch relevance scoring process.

[0131] Figure 6 A flowchart illustrating visual patch relevance scoring and order-preserving screening according to an embodiment of this application is shown.

[0132] In one specific implementation, such as Figure 6 As shown, the visual patch relevance scoring and spatial order-preserving screening process is used to infer the intent based on the output of the multi-agent intent analysis submodule. The correlation between each visual patch in the input image and the inferred intent is calculated, and the visual patches are filtered based on the correlation score. While maintaining the original spatial order of the selected visual patches, the filtered visual patch embedding sequence is extracted from the original visual patch embedding sequence output by the original multimodal large model visual tower. The processing object is the input image. Inference of intent The patch relevance scoring unit loads an auxiliary visual-text alignment model, which can be CLIP (Contrastive Language-Image Pre-training, e.g., batch-contrastive image-text semantic alignment pre-training model), SigLIP (Sigmoid Loss for Language Image Pre-training, an image-text pre-training model that uses Sigmoid pairwise loss and does not rely on batch global information, e.g., pairwise decision-based image-text semantic alignment pre-training model), or ALIGN (A Large-scale Image and Noisy-textembedding, a large-scale noisy image-text embedding alignment pre-training model). The model is used to extract visual patch block-level features and infer intent text features, respectively.

[0133] First, the auxiliary visual encoder processes the input image. Encode the data and obtain all visual patch block-level features according to the visual patch division method corresponding to the input image. Assume the input image corresponds to... If there are 1 visual patch, then the visual patch block-level feature sequence output by the auxiliary visual encoder can be represented as: ,in, This represents the visual patch block-level feature sequence output by the auxiliary visual encoder. Indicates the first Each visual patch corresponds to an auxiliary visual feature. The range of values ​​is , This represents the total number of visual patches corresponding to the input image. (A positive integer greater than or equal to 2).

[0134] Secondly, the auxiliary text encoder infers intent. Encoding yields the text feature vector corresponding to the inferred intent. To reduce the impact of feature magnitude differences on the relevance calculation results, L2 normalization is performed on the text feature vector to obtain the normalized guiding text vector. , represented as: = .in, This illustrates the auxiliary text encoder in the auxiliary vision-text alignment sub-model.

[0135] Furthermore, L2 normalization is performed on each visual patch block-level feature output by the auxiliary visual encoder. For the ... Each visual patch, its normalized visual patch-level features. It can be represented as: ;in, Indicates the first Each visual patch corresponds to an auxiliary visual feature. This represents the corresponding normalized visual patch-level features.

[0136] Obtain the normalized guiding text vector and normalized visual patch-level features Then, the cosine similarity between the two is calculated to obtain the first... Relevance score for each visual patch , represented as: ,in, , The range of values ​​is .in, Indicates the first The semantic relevance between a visual patch and the multi-agent inferred intent. Relevance score. The larger the value, the higher the degree of matching between the visual semantics contained in the corresponding visual patch and the core semantics of the image represented by the multi-agent inference intent; the higher the relevance score. The smaller the value, the lower the degree of matching between the corresponding visual patch and the core semantics of the image. This is achieved by calculating all... The relevance scores of each visual patch can be used to obtain a visual patch relevance score sequence: .

[0137] In one alternative implementation, to facilitate a unified scale representation of the relevance of different visual patches, the visual patch relevance score sequence can be further normalized based on a temperature parameter. For the... Each visual patch, with its normalized relevance weights. It can be represented as: .

[0138] in, Indicates the first Normalized relevance weights corresponding to each visual patch The temperature parameter, representing a value greater than zero, is used to adjust the concentration of the correlation weight distribution among different visual patches. The normalized correlation weights corresponding to each visual patch satisfy: ,as well as: .

[0139] Because in In the case of the exponential mapping function Regarding the correlation score As a strictly monotonically increasing function, therefore, for the normalized correlation weights Sort by high to low, compared to directly using the original relevance scores. Sort the visual patches in descending order of their index order, i.e.: .

[0140] Therefore, the temperature normalization process described above is only used to map the relevance scores of different visual patches to normalized relevance weights with a uniform scale. It does not change the original relevance ranking of visual patches, nor does it change the candidate visual patch set determined by the subsequent Top-K index filtering. In the specific implementation, the relevance scores can be directly used as the basis for the algorithm. Subsequent Top-K index filtering operations can also be performed based on normalized relevance weights consistent with their ranking relationship. Perform the index filtering operation.

[0141] It should be noted that the output of the auxiliary visual encoder and normalized visual patch-level features These are only used to calculate the visual patch relevance score and determine the visual patch index, and are not directly used as input to the multimodal projector of the original multimodal large model. The visual features that are actually fed into the multimodal projector are still extracted from the original visual patch embedding sequence output by the visual tower of the original multimodal large model according to the determined index, thereby separating the visual patch relevance evaluation process from the downstream visual feature source.

[0142] Compared to existing technologies, this application uses image-derived inference intent as query text for relevance calculation, rather than the original user question; in order to establish a screening criterion based on the core semantics of the image, normal scene visual areas are retained first, reducing the probability of local patches with embedded harmful instructions being selected.

[0143] According to an embodiment of this application, the above-mentioned index filtering operation and spatial order-preserving rearrangement operation performed on the original visual patch embedding sequence using the visual patch relevance score sequence to obtain the filtered visual patch embedding sequence includes: filtering the visual patch relevance score sequence based on descending index to obtain a candidate index set; sorting the candidate index set in ascending order according to the spatial position of the visual patch embedding in the input image to obtain an order-preserving index set; and filtering the original visual patch embedding sequence according to the order-preserving index set to obtain the filtered visual patch embedding sequence.

[0144] The above embodiments involve visual patch index filtering and spatial order-preserving rearrangement.

[0145] In one specific implementation, the auxiliary visual-text alignment sub-model is only used to generate visual patch block-level features, guide text embedding, and calculate relevance scores, and does not directly use the visual features it generates as input to the downstream modules of the original multimodal large model.

[0146] Based on the obtained visual patch relevance score sequence Perform a Top-K selection operation, selecting the highest-scoring candidates based on their relevance scores from highest to lowest. ( The position indices of the visual patches (where each index is a positive integer) are used to form a sequence like this: Figure 6 The candidate index set shown ,in, Indicates the number of visual patches retained, and satisfies ; This represents the set of candidate indices obtained based on visual patch relevance scores. Thus, from the original... Retain in each visual patch One visual patch, the rest The visual patch will not be included in the subsequent multimodal fusion process.

[0147] It should be noted that the Top-K index selection operation determines candidate visual patches based on their relevance scores. The resulting candidate index order reflects the ranking of relevance scores and does not necessarily correspond to the spatial order of the corresponding visual patches in the original input image. If candidate visual patches are directly input into the subsequent multimodal projector according to their relevance score order, the original spatial order among the retained visual patches may be altered. Therefore, after obtaining the candidate index set... Then, spatially ordered sorting is performed on the candidate index set.

[0148] In one alternative implementation, the spatially ordered sorting process can be formally represented using the two-dimensional spatial coordinates of each visual patch within the corresponding image patch grid of the input image. Let the... The spatial location of each visual patch in the input image is represented as follows: ,in, Indicates the first The spatial location corresponding to each visual patch This indicates the row position of the visual patch within the image patch grid. This indicates the column position of the visual patch within the image patch grid.

[0149] For an implementation that arranges and numbers visual patches in a row-first manner, the order of any two visual patches in terms of spatial position can be defined as follows: ,in, This indicates the spatial order of visual patches in the original input image. When two visual patches are located in different image patch rows, the visual patch in the earlier image patch row has a higher spatial order; when two visual patches are located in the same image patch row, the visual patch in the earlier image patch column has a higher spatial order.

[0150] Based on the aforementioned spatial order, the candidate index set... The visual patch indexes in the data are reordered to obtain the following: Figure 6 The set of order-preserving indexes shown: ,in, This represents the set of indices after spatially ordered sorting. Represents the first in the pre-order index set A visual patch index, and satisfying: When the visual patches of the original multimodal large model are numbered consecutively in row-major order, the above spatial ordering relation can also be equivalently represented as: .

[0151] Through the spatially ordered sorting described above, the Top-K index selection operation only determines which visual patches are retained, without altering the spatial order of the retained visual patches in the subsequent visual patch sequence based on their relevance scores. In other words, the relevance score is used to determine the set of visual patches, while the spatial location information is used to determine the order of the selected visual patches, ensuring that the filtered visual patches maintain their relative spatial order consistent with their original input image.

[0152] Therefore, this application reduces the number of visual patches entering the subsequent multimodal fusion process while avoiding spatial order disturbances caused by directly rearranging visual patches according to their relevance scores. It should be noted that the spatial order preservation operation maintains the spatial order relationship between the selected visual patches relative to the original input image, rather than regenerating or modifying the visual features output by the original multimodal large-scale model visual tower. This allows for seamless integration with the subsequent process of extracting visual patch embeddings from the original visual patch embedding sequence according to the order-preserving index.

[0153] On the other hand, input image Encoding the original multimodal large model visual tower yields the original visual patch embedding sequence containing N visual patch embeddings: ,in, This represents the original visual patch embedding sequence output by the original multimodal large model visual tower. Indicates the first An original visual patch is embedded. This indicates the feature dimension embedded in the original visual patch.

[0154] The visual patch corresponding to the auxiliary visual encoder and the visual patch corresponding to the original multimodal large model visual tower are established according to their spatial positions in the input image, so that the patch index determined by the auxiliary visual-text alignment model can be used to locate the corresponding visual patch embedded in the output of the original visual tower.

[0155] Based on the order-preserving index set From the original visual patch embedding sequence Extract the original visual patch embeddings at the corresponding positions to obtain the filtered visual patch embedding sequence: ,in, This indicates the original index extracted from the original visual patch embedding sequence output by the original multimodal large model visual tower. Visual patch embedding, This represents the first index in the ordered index set obtained after relevance filtering and spatial order-preserving sorting. A visual patch index, This indicates the position number in the filtered visual patch sequence.

[0156] in, This represents the filtered visual patch embedding sequence, which contains A visual patch is embedded. It should be noted that... Each visual patch embedded in the model is directly derived from the output of the original multimodal large model visual tower. Patch-level features output by the auxiliary visual encoder and the features obtained by normalization. It is used only for visual patch relevance scoring and patch index determination, and is not directly input into the multimodal projector of the original multimodal large model. In this way, the visual patch selection decision is separated from the downstream visual feature source. While using the auxiliary visual-text alignment sub-model to complete the relevance evaluation, it ensures that the visual features that actually enter the multimodal projector still come from the original feature space of the visual tower of the original multimodal large model.

[0157] The specific implementation described above involves forcibly restoring the original spatial order after screening, reusing only the auxiliary model for scoring and indexing decisions, and taking all downstream input features from the native embeddings of the visual tower without using the visual features of the auxiliary model. Through this process, the relevance evaluation and indexing decisions of visual patches are separated from the source of downstream features of the visual patches. This utilizes an auxiliary visual-text alignment model to complete the screening of visual patches based on inferred intent, while maintaining that downstream visual features originate from the native feature space of the original multimodal large model's visual tower. Simultaneously, spatial order preservation maintains the original spatial positional relationships between the selected visual patches, thereby reducing the impact of the visual patch screening process on the normal visual understanding ability of the original multimodal large model.

[0158] According to embodiments of this application, the above-mentioned multimodal fusion processing of the selected visual patch embedding sequence and the original user question using a multimodal large model to generate a safe answer that filters out risk information includes: using the multimodal projector of the multimodal large model to map the selected visual patch embedding sequence to the language model feature space to obtain the projected visual patch embedding sequence; performing multimodal fusion processing on the projected visual patch embedding sequence and the original user question; and based on the result of the multimodal fusion processing, using the language model autoregression of the multimodal large model to generate a safe answer that filters out risk information.

[0159] The above embodiments involve multimodal fusion and response generation.

[0160] In one specific implementation, the object of processing is the filtered visual patch embedding sequence. Original user issues The multimodal projection and response generation unit performs the following operations: First, it... First, the input is fed into the original MLLM multimodal projector and mapped to the language model feature space; second, the projected visual features are compared with the original user question. The data are fed into the language model together; third, the language model generates the final answer through autoregression. : .in For causal language models, This is a multimodal projector. The key constraint is inferring intent. This input is not included in this step. The beneficial effect is that, while filtering harmful visual information, the model ensures its responses revolve around the user's original question, balancing security and usability. Specifically, the visual patch relevance score is used to determine which visual patches need to be retained, while spatial ordering is used to determine the location of the retained visual patches. The order of arrangement in the text. Therefore, The visual features are not regenerated by the auxiliary visual encoder, but are based on the order-preserving index. The original visual patch embeddings at corresponding positions are extracted from the original visual patch embedding sequence output by the original multimodal large model visual tower. Thus, the auxiliary visual-text alignment sub-model is only used for relevance scoring and index determination, and the visual features actually input to the multimodal projector still come from the visual tower of the original multimodal large model.

[0161] Figure 7 A structural diagram of a multimodal large-scale defense device based on multi-agent intent analysis according to an embodiment of this application is shown.

[0162] like Figure 7As shown, the above-mentioned multimodal large model defense device based on multi-agent intent analysis includes an input acquisition unit 1, a visual tower encoding unit 2, a multi-agent intent analysis unit 3, a patch relevance scoring unit 4, a visual patch filtering unit 5, and a multimodal projection and response generation unit 6.

[0163] Input acquisition unit 1 is used to acquire the input image and the original user question.

[0164] Visual tower encoding unit 2 is used to encode the input image using a multimodal large model visual tower encoder to obtain the original visual patch embedding sequence.

[0165] The multi-agent intent analysis unit 3 is used to perform global semantic parsing and content recognition on the input image using an image understanding agent to obtain multi-dimensional semantic information of the input image. Based on the multi-dimensional semantic information, a global image description text is generated. A phrase cue extraction agent is used to perform local text identifier detection on the input image to obtain phrase cues. Finally, an intent inference agent is used to fuse the global image description text and phrase cues to obtain the multi-agent inferred intent of the input image.

[0166] Patch relevance scoring unit 4 is used to score the relevance between each visual patch in the original visual patch embedding sequence and the multi-agent inference intent using the auxiliary visual-text alignment sub-model of the multimodal large model, and obtain the visual patch relevance score sequence.

[0167] The visual patch filtering unit 5 is used to perform index filtering and spatial order-preserving rearrangement operations on the original visual patch embedding sequence using the visual patch correlation score sequence to obtain the filtered visual patch embedding sequence.

[0168] The multimodal projection and response generation unit 6 is used to perform multimodal fusion processing on the filtered visual patch embedding sequence and the original user question using a multimodal large model to generate a safe response that filters out risky information.

[0169] in, Figure 7 The defense device shown is deployed between the MLLM vision tower and the multimodal projector. The various components of the device and their interactions are as follows: Figure 7As shown. The input acquisition unit receives the external input image and the original user question at its input end, and its output ends are connected to the visual tower encoding unit and the multi-agent intent analysis unit, respectively. Its function is to receive and distribute task data. The visual tower encoding unit receives the input image at its input end, internally calls MLLM to freeze the visual tower, and outputs the original visual patch embedding sequence, which is connected to the visual patch filtering unit. The multi-agent intent analysis unit is the core distinguishing module of this application. It receives the input image at its input end and internally integrates an image understanding agent, a phrase cue extraction agent, and an intent inference agent. The agent links analyze semantic information, and the output end outputs the inferred intent. The system connects to a patch relevance scoring unit. This unit receives the input image and the inferred intent, incorporates an auxiliary visual-text alignment model, and outputs a sequence of visual patch relevance scores, which connects to a visual patch filtering unit. The visual patch filtering unit receives two inputs: the original visual patch embedding sequence and the patch relevance score. Internally, it performs Top-K index selection, index-preserving sorting, and original feature extraction, outputting a filtered visual patch embedding sequence, which connects to a multimodal projection and response generation unit. This unit receives two inputs: the filtered visual patch embedding sequence and the original user question, and outputs the final model response. The internal data flow interaction logic of the device is serial across all modules, eliminating feature loops. The original MLLM backbone network parameters are frozen throughout the process; each unit only adds forward inference logic and does not interfere with the model training process.

[0170] In one specific implementation, the defense scheme provided in this application is deployed between the visual tower and the multimodal projector of the MLLM. The parameters of the original MLLM's visual tower, multimodal projector, and language model are frozen throughout the process, and no retraining or fine-tuning is performed. The device as a whole includes an input acquisition unit 1, a visual tower encoding unit 2, a multi-agent intent analysis unit 3, a patch relevance scoring unit 4, a visual patch filtering unit 5, and a multimodal projection and response generation unit 6. Each unit interacts sequentially, and data flow between units is unidirectional. The overall implementation logic is as follows: receiving the input image and the original user question; on the one hand, obtaining the original visual patch embedding sequence through the MLLM's built-in visual tower encoding; on the other hand, activating the multi-agent intent analysis unit to generate inferred intent based on the image content; using an auxiliary visual-text alignment sub-model to calculate the relevance between each visual patch and the inferred intent, filtering highly relevant visual patch indices while maintaining the original spatial order; extracting features from the original visual patch embedding sequence based on the indices and sending them to the downstream multimodal projector; and combining the unmodified original user question with the language model to generate a response. By filtering visual patches that are irrelevant to the core semantics of the image and carry harmful information before multimodal fusion, harmful visual input defense is achieved.

[0171] This application is not limited to the specific embodiments described above. Without departing from the technical concept of this application, each functional module and processing strategy can be adjusted or replaced according to the actual application scenario. For example, in terms of multi-agent deployment, in addition to using local serial inference for each agent, image understanding agents and intent inference agents can be deployed on different computing nodes, and information interaction between agents can be achieved through message queues or other communication mechanisms to adapt to the deployment requirements of large-scale multimodal inference clusters in the cloud. Regarding relevance scoring, in addition to using pre-trained static CLIP, SigLIP, and other models, the auxiliary visual-text alignment model can also use a graph-text alignment model fine-tuned with domain-specific data, or a lightweight graph-text dual-tower model trained based on contrastive learning, thereby reducing inference latency while ensuring relevance evaluation capabilities.

[0172] For the visual patch selection process, in addition to using a fixed Top-K selection method that maintains the original spatial order, a relevance score threshold can be preset. Only visual patches with a relevance score higher than the threshold are retained, and further sorted according to their spatial location to improve the adaptability of the solution to different image content distributions and open graphic interaction scenarios. Regarding the generation of inferred intent, in addition to generating inferred intent in real time during the inference stage, corresponding inferred intents can be pre-generated offline for frequently occurring images or typical application scenarios, and a caching mechanism can be established. When the cache is hit during inference, the corresponding inferred intent is directly read. This reduces the computational overhead of multi-agent collaborative reasoning and improves the overall reasoning throughput.

[0173] Furthermore, the number of visual patches retained, K, can be set not only as a fixed value or a fixed proportion, but also dynamically adjusted in conjunction with a risk assessment mechanism. For example, a risk assessment submodule can be constructed to determine the corresponding risk level based on whether the image contains text information, suspicious patterns, or other preset risk characteristics, and adaptively adjust the value of K accordingly. This allows K to be appropriately reduced in higher-risk scenarios, thereby enhancing the ability to suppress potentially malicious visual information. In addition, the deployment location of the defense module described in this application is not limited to between the visual encoder output and the multimodal projector. It can also be encapsulated as an independent inference middleware and interfaced with existing model systems through a standardized multimodal large language model inference interface to achieve an external, low-intrusion defense deployment, thereby implementing corresponding security protection functions without modifying the original model inference chain code.

[0174] Before multimodal fusion, this application uses the inferred intent obtained from the semantic inference of the image itself as a guide to perform fine-grained screening on the patch embedding sequence output by the visual tower, achieving the following technical effects.

[0175] (1) Using image ontology information to generate inference intent as a screening guide, the screening benchmark is removed from the input text that the attacker can control.

[0176] This application sets up three types of collaborative agents: an image understanding agent, a phrase cue extraction agent, and an intent inference agent. The image understanding agent performs visual semantic abstraction on the input image to obtain image description text. The phrase cue extraction agent identifies local text and symbols in the image to obtain phrase cues. The intent inference agent integrates the two abstractions to obtain the inferred intent that represents the core semantics of the image. This inferred intent serves as the sole query text for subsequent patch relevance calculations, and the original user question does not participate in visual filtering.

[0177] The guidance information from the visual filtering determines which patches are retained. The original user question is entirely controllable by the attacker; using it as the query text is equivalent to handing over the filtering power to the attack surface itself. The inferred intent, however, is derived solely from the image content. If an attacker wants to manipulate the inferred intent, they can only modify the image content itself. Modifying the image content will simultaneously change the image description text, thus limiting their manipulation space to the self-consistency of the image's semantics. The intermediate results generated by the division of labor among the three types of agents are all natural language text, which can be audited and verified.

[0178] This application makes the criteria for patch selection based on the true semantics of the image rather than user questions, avoiding interference from leading words, jailbreaking instructions, and format requirements in user questions on the selection results. It also reduces the risk that attackers may use textual inducement to preferentially retain local patches with steganographic harmful instructions, and overcomes the defect that the selection criteria can be manipulated.

[0179] (2) The intent analysis path and the answer generation path are decoupled, and the defense intervention does not change the user's original question.

[0180] The inferred intent is transmitted only to the patch relevance scoring unit and is not sent to the multimodal projector and language model; the final answer generation always uses the original user question without any rewriting, and this application does not output security or harmful labels or perform request interception.

[0181] Existing solutions apply defensive actions at the request level; if a false positive occurs in the risk assessment, the user request is rejected entirely. In contrast, the defensive actions in this application only apply to the visual evidence fed into the language model, ensuring that the language model always receives the user's genuine request. Even if a patch is mistakenly removed, the model will still respond to the user's question based on the remaining patches; the failure mode manifests as a reduction in the detail of the response rather than service interruption.

[0182] This application avoids the problem of normal requests being blocked as a whole due to false positives in binary decision-making, ensuring that the output content always matches the user's real needs, and maintaining the interactive usability of the model while improving security.

[0183] (3) Using visual patches as the smallest processing unit, fine-grained filtering of harmful visual information is achieved before multimodal fusion.

[0184] This application embeds the defense process between the visual tower and the multimodal projector. The object of the treatment is the original visual patch embedding sequence output by the visual tower. The correlation score between each patch and the inferred intent is calculated, and the patch with the highest score is selected. One patch will remove patches that are irrelevant to the core semantics of the image before they enter the multimodal projector.

[0185] Harmful instructions steganographically embedded in localized areas of an image are diluted by numerous normal region features during whole-image risk scoring, making them difficult to capture by the input image detection. Furthermore, at the patch granularity, these regions have a significantly lower correlation with the core semantics of the image than normal scene regions, ranking low in relevance and thus naturally being eliminated in the Top-K selection process. The defense mechanism, deployed after the visual tower, transforms the target from the unparseable raw image file into a structured patch embedding sequence, a prerequisite for achieving local elimination.

[0186] This application enables the removal of harmful visual information before it is integrated with the text, overcoming the shortcomings of the whole-image-level processing granularity that leads to the failure to detect local steganographic content, while eliminating the need for a binary decision of allowing or blocking the entire image.

[0187] (4) The auxiliary alignment model only undertakes index decision-making, and the downstream features are taken from the original embedding of the visual tower and rearranged according to the original spatial order.

[0188] This application uses an auxiliary vision-text alignment model to calculate the relevance score and determine the patch index, but all features fed into the multimodal projector are extracted from the original visual patch embedding sequence output by the vision tower based on this index; the candidate index set is sorted in ascending order according to the spatial position of the original patch image before extraction to form an order-preserving index set.

[0189] Scoring and feature extraction are two separable actions. The feature space of the auxiliary alignment model is not consistent with the feature space of the MLLM visual tower. If the visual features of the auxiliary model are directly fed downstream, the multimodal projector will receive input that does not match its training distribution. However, if only the relevance ranking results of the auxiliary model are borrowed and the features are still taken from the visual tower, then every feature vector obtained downstream will be exactly the same as when this application was not deployed. This application only reduces, rather than replaces, the feature sequence. At the same time, the order of the patch embedding sequence encodes the two-dimensional spatial positional relationship of the image. Sorting by score will invalidate this correspondence, so the ascending index must be restored before extraction.

[0190] This application neither causes a shift in the distribution of visual features nor destroys the spatial topological information of the image, thus preserving the original model's capabilities in normal text-image question-answering tasks such as counting, localization, and layout understanding to the greatest extent possible, overcoming the defect that the filtering process damages the model's normal capabilities.

[0191] (5) Freeze the backbone network parameters throughout the process and deploy in a low-intrusion manner.

[0192] All processing logic in this application resides in the forward inference link between the visual tower and the multimodal projector. The visual tower, multimodal projector, and language model parameters of the original MLLM are frozen throughout the process, and no retraining or fine-tuning is performed. This defense process can be further encapsulated as an independent inference middleware and interfaced with existing model systems through a standardized inference interface.

[0193] The external capabilities required by this application are only text reasoning for several intelligent agents and a lightweight image-text alignment model. Both are accessed by invocation and are not coupled with the parameters of the protected model. When the basic model is changed, it is only necessary to ensure that the interface form of its visual tower output patch embedding sequence remains unchanged, and the defense logic can be directly reused.

[0194] This application avoids the performance regression and retraining costs of general tasks caused by security fine-tuning, and can be deployed as an add-on on the already launched inference service, overcoming the shortcomings of high deployment cost and poor portability.

[0195] Figure 8 A block diagram of an electronic device suitable for implementing a multimodal large model defense method based on multi-agent intent analysis, according to an embodiment of this application, is shown.

[0196] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage portion 808 into a random access memory 803. The processor 801 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0197] Random access memory 803 stores various programs and data required for the operation of electronic device 800. Processor 801, read-only memory 802, and random access memory 803 are interconnected via bus 804. Processor 801 executes various steps of the method flow according to embodiments of this application by executing programs stored in read-only memory 802 and / or random access memory 803. It should be noted that the programs may also be stored in one or more memories other than read-only memory 802 and random access memory 803. Processor 801 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0198] According to embodiments of this application, the electronic device 800 may further include an input / output interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card, such as a local area network card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0199] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0200] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 802 described above, and / or random access memory 803, and / or one or more memories other than read-only memory 802 and random access memory 803.

[0201] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0202] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0203] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0204] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A multimodal large-scale model defense method based on multi-agent intent analysis, characterized in that, The method includes: The input image is encoded using a multimodal large model visual tower encoder to obtain the original visual patch embedding sequence; An image understanding agent is used to perform global semantic parsing and content recognition on the input image to obtain multi-dimensional semantic information of the input image. Based on the multi-dimensional semantic information, a global image description text is generated. A phrase cue extraction agent is used to perform local text identifier detection on the input image to obtain phrase cues. Finally, an intent inference agent is used to fuse the global image description text and the phrase cues to obtain the multi-agent inferred intent of the input image. The auxiliary visual-text alignment sub-model of the multimodal large model is used to score the relevance between each visual patch in the original visual patch embedding sequence and the inferred intent of the multi-agent, resulting in a visual patch relevance score sequence. The original visual patch embedding sequence is subjected to index filtering and spatial order-preserving rearrangement operations using the visual patch relevance score sequence to obtain the filtered visual patch embedding sequence. The multimodal large model is used to perform multimodal fusion processing on the filtered visual patch embedding sequence and the original user question to generate a safe answer that filters out risk information.

2. The method according to claim 1, characterized in that, The input image is encoded using a multimodal large-scale visual tower encoder to obtain the original visual patch embedding sequence, which includes: The input image is preprocessed and its pixel values ​​are normalized to obtain a standard input image; The standard input image is divided into multiple non-overlapping image blocks; The pixel vectors in multiple non-overlapping image blocks are mapped to feature vectors with a preset dimension to obtain multiple initial visual patch embeddings; Add positional encoding to each initial visual patch embedding to obtain multiple visual patch embeddings with spatial positional information; For each of the visual patch embeddings with spatial location information, a feature update operation and a semantic association operation based on a self-attention interaction mechanism are performed to obtain multiple updated visual patch embeddings. The multiple updated visual patch embeddings are normalized to obtain the original visual patch embedding sequence.

3. The method according to claim 1, characterized in that, Using an image understanding agent to perform global semantic analysis and content recognition on the input image, multi-dimensional semantic information of the input image is obtained. Based on this multi-dimensional semantic information, a global image description text is generated, including: The input image is subjected to image size normalization and pixel value standardization to obtain a preprocessed input image; Visual features are extracted from the preprocessed input image to obtain the image visual features; Based on the visual features of the image, target object detection, scene classification, person detection, human action recognition and text content recognition operations are performed on the preprocessed input image to obtain object information, scene information, person information, action information and image embedded text information. The object information, scene information, character information, action information, and image embedded text information are integrated into structured multi-dimensional semantic information, and image global description text representing the global content of the input image is generated based on the multi-dimensional semantic information.

4. The method according to claim 1, characterized in that, The phrase cue extraction agent performs local text label detection on the input image to obtain phrase cues, including: The local text, local identifiers, and / or local screenshot text information of the input image are filtered to obtain the local text information filtering results; If the local text information filtering result is a valid local text that meets the preset reference value, the local text information filtering result is subjected to text feature extraction to obtain the phrase clue; If the results of the local text information filtering are valid local texts that do not meet the preset reference value, the phrase clues will be left blank.

5. The method according to claim 4, characterized in that, The multi-agent inference agent fuses the global description text of the image and the phrase cues to obtain the multi-agent inferred intent of the input image, including: When the phrase clues are empty, semantic reasoning is performed on the global image description text to obtain the multi-agent inference intent; When the phrase cue is not empty, the global description text of the image and the phrase cue are fused, and semantic reasoning is performed on the fusion result to obtain the multi-agent inference intent.

6. The method according to claim 1, characterized in that, Using the auxiliary visual-text alignment sub-model of the multimodal large model, the relevance scores between each visual patch in the original visual patch embedding sequence and the multi-agent inferred intent are calculated, resulting in a visual patch relevance score sequence including: The input image is encoded using the auxiliary visual encoder of the auxiliary visual-text alignment sub-model to obtain a visual patch-level feature sequence corresponding to the visual patch in the original visual patch embedding sequence; The auxiliary text encoder of the auxiliary vision-text alignment sub-model is used to encode the multi-agent inferred intent to obtain a text vector; The visual patch-level feature sequence is normalized to obtain a normalized visual patch-level feature sequence. Calculate the similarity between the text vector and each normalized visual patch-level feature in the normalized visual patch-level feature sequence to obtain the visual patch relevance score sequence.

7. The method according to claim 6, characterized in that, The auxiliary visual-text alignment sub-model includes a batch comparison-based image-text semantic alignment pre-training sub-model, a pairwise decision-based image-text semantic alignment pre-training model, or a noisy image-text embedding alignment pre-training model. The similarity includes cosine similarity, Euclidean distance similarity, and inner product similarity.

8. The method according to claim 1, characterized in that, Using the visual patch relevance score sequence, an index filtering operation and a spatial order-preserving rearrangement operation are performed on the original visual patch embedding sequence to obtain the filtered visual patch embedding sequence, which includes: The visual patch relevance score sequence is filtered based on descending index to obtain a candidate index set; The candidate index set is sorted in ascending order according to the spatial position of the visual patch embedded in the input image to obtain the order-preserving index set. The original visual patch embedding sequence is filtered according to the order-preserving index set to obtain the filtered visual patch embedding sequence.

9. The method according to claim 1, characterized in that, The multimodal large model is used to perform multimodal fusion processing on the filtered visual patch embedding sequence and the original user question to generate a safe answer that filters out risk information, including: The selected visual patches are embedded into the feature space of the sequence mapping language model using the multimodal large model's multimodal projector to obtain the projected visual patch embedding sequence. The projected visual patch embedding sequence and the original user question are subjected to multimodal fusion processing. Based on the result of the multimodal fusion processing, the language model of the multimodal large model is used to generate the safe answer that filters out the risk information through autoregression.

10. A multimodal large-scale defense device based on multi-agent intent analysis, characterized in that, The device includes: The input acquisition unit is used to acquire the input image and the original user question; The visual tower encoding unit is used to encode the input image using a multimodal large model visual tower encoder to obtain the original visual patch embedding sequence; The multi-agent intent analysis unit is used to perform global semantic parsing and content recognition on the input image using an image understanding agent to obtain multi-dimensional semantic information of the input image, generate global image description text based on the multi-dimensional semantic information, perform local text identifier detection on the input image using a phrase cue extraction agent to obtain phrase cue, and fuse the global image description text and the phrase cue using an intent inference agent to obtain the multi-agent inferred intent of the input image. The patch relevance scoring unit is used to use the auxiliary visual-text alignment sub-model of the multimodal large model to score the relevance between each visual patch in the original visual patch embedding sequence and the inference intent of the multi-agent, so as to obtain a visual patch relevance score sequence. The visual patch filtering unit is used to perform index filtering and spatial order-preserving rearrangement operations on the original visual patch embedding sequence using the visual patch relevance score sequence to obtain the filtered visual patch embedding sequence. The multimodal projection and response generation unit is used to perform multimodal fusion processing on the filtered visual patch embedding sequence and the original user question using the multimodal large model to generate a safe response that filters out risk information.