A cross-modal agent security reasoning method and system based on permission calibration

Abstract

The application discloses a cross-modal intelligent agent security reasoning method and system based on permission calibration, which comprises the following steps: acquiring an image of a screen and window metadata; determining the permission identification of each window according to the window metadata, and calibrating each visual block in the image according to the permission identification to obtain a permission visual block; encoding the permission visual block through a visual encoder to generate a visual feature vector, and generating a privilege feature vector according to the visual feature vector; predicting the control intention probability through an intention classification head according to the privilege feature vector; generating a privilege attention mask matrix according to the privilege feature vector and the control intention probability, and performing permission restriction on the model reasoning process according to the privilege attention mask matrix to obtain a safe reasoning result. The application can effectively inhibit the interference of malicious control instructions on the cross-modal intelligent agent reasoning process, ensure that the legal business information can normally participate in the model reasoning, and improve the security and reliability of the graphical user interface interaction process.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and computer technology, and in particular relates to a cross-modal intelligent agent secure reasoning method and system based on permission marking. Background Technology

[0002] With the development of large language models and visual language models, cross-modal intelligent agents can combine image and text information to complete the recognition, understanding, and operation of graphical user interfaces, and have been widely used in fields such as office automation, intelligent assistants, and human-computer interaction. In these applications, the intelligent agent typically performs interface analysis and operational decisions based on screen images and user input, and its execution results directly affect the accuracy and security of the system operation.

[0003] In related technologies, the security of intelligent agents is mainly improved through methods such as prompt word filtering, content moderation, security policy configuration, and model training optimization, which can reduce the impact of malicious input on the model to a certain extent. However, in complex graphical user interface environments, information from different sources often exists on the same interface simultaneously. Related technologies struggle to accurately identify and distinguish the credibility of different information, making the intelligent agent susceptible to interference from untrusted information during reasoning, thereby affecting the reliability of model reasoning results and the security of system operation. Summary of the Invention

[0004] The embodiments of this application provide a cross-modal intelligent agent secure reasoning method and system based on permission labeling, which can effectively suppress the interference of malicious control commands on the cross-modal intelligent agent reasoning process, and ensure that legitimate business information can participate in model reasoning normally, thereby improving the security and reliability of the cross-modal intelligent agent graphical user interface interaction process.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to a first aspect of the embodiments of this application, a cross-modal intelligent agent secure reasoning method based on permission identification is provided, the method comprising:

[0007] Retrieve the image data and window metadata corresponding to the current physical screen;

[0008] The window permission identifier of each window in the image data is determined based on the window metadata, and the visual blocks in the image data are labeled based on the window permission identifier to obtain permission visual blocks;

[0009] The permission visual patch is encoded using a visual encoder to generate a visual feature vector, and a privileged feature vector is generated based on the visual feature vector.

[0010] The privileged feature vector is classified and predicted using the intent classification head to obtain the control intent probability;

[0011] A privileged attention masking matrix is ​​generated based on the privileged feature vector and the control intent probability. The privileged attention masking matrix is ​​then used to impose permission constraints on the model inference process, resulting in a secure inference result with permission constraints.

[0012] According to a second aspect of the embodiments of this application, a cross-modal intelligent agent secure reasoning system based on permission labeling is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the various steps of the cross-modal intelligent agent secure reasoning method based on permission labeling described above.

[0013] In embodiments of this application, the application acquires image data and window metadata corresponding to the current physical screen; determines the window permission identifier of each window in the image data based on the window metadata, and calibrates each visual patch in the image data based on the window permission identifier to obtain permission visual patches; encodes the permission visual patches using a visual encoder to generate visual feature vectors, and generates privileged feature vectors based on the visual feature vectors; classifies and predicts the control intent probability using an intent classification head using the privileged feature vectors; generates a privileged attention masking matrix based on the privileged feature vectors and the control intent probability, and applies permission constraints to the model inference process based on the privileged attention masking matrix to obtain a secure inference result with permission constraints.

[0014] Therefore, this application acquires image data and window metadata of the current physical screen, uses the window metadata to determine the window permission identifier of each window, and further maps the window permission identifier to visual tiles, so that the visual tiles have permission attributes corresponding to their information sources before entering visual encoding and model inference. On this basis, the visual feature vector is combined with permission information to form a privileged feature vector, and a privileged attention mask matrix is ​​generated by combining the control intent probability, so that the permission attributes corresponding to the window source can be gradually passed from the image space layer to the model feature layer and attention calculation layer. As a result, when the model performs cross-modal inference, it no longer treats all visual information in the physical screen as equally trustworthy information, but can apply differentiated attention constraints to different visual information according to the source permission and control intent of the visual information, reducing the impact of visual content in low-permission or untrusted windows on the model inference process, thereby improving the reliability of cross-modal intelligent agent inference results and the security of operational decisions in complex graphical user interface environments.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0017] Figure 1 A flowchart of a cross-modal intelligent agent secure reasoning method based on permission identification according to this application is shown;

[0018] Figure 2 The diagram illustrates the data interaction of a cross-modal intelligent agent secure reasoning method based on permission labeling according to this application.

[0019] Figure 3 The diagram illustrates the architecture of a cross-modal intelligent agent secure reasoning system based on permission labeling, as proposed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0024] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0025] To enable those skilled in the art to better understand this application, the important terms involved in this application will be briefly explained first.

[0026] Cross-modal intelligent agents: intelligent agents that can acquire and process information from different modalities such as images and text, and perform joint reasoning based on information from different modalities to complete interface content understanding, information extraction and operation decision-making.

[0027] Graphical User Interface (GUI): A visual interactive interface provided by the operating system or application and displayed on the physical screen. It may include windows, text, icons, buttons, input boxes, web page content, and other visual interface elements.

[0028] Physical screen: The display area where the operating system is currently outputting graphics.

[0029] A window is a graphical user interface object managed by the operating system's window management mechanism and occupying a corresponding display area on the physical screen. Different windows can correspond to different applications, system components, web pages, or other user interface content.

[0030] Access Level: A hierarchical attribute used to characterize the relative access relationships between different information sources during model inference. High access and low access are used to represent the relative access hierarchy between different information sources, without limiting the specific number of access levels or the encoding method.

[0031] Token: A discrete input unit or a feature representation corresponding to that input unit used by the model when processing input information. In this application, visual patches, after visual encoding and corresponding feature processing, can form visual tokens that participate in subsequent calculations of cross-modal large models.

[0032] A visual encoder is a neural network module that converts an input image or visual patch into a visual feature representation that the model can process. The visual encoder extracts features from the pixel information in the visual patch, generating a visual feature vector corresponding to each visual patch.

[0033] Latent space: A high-dimensional feature space used within the model to represent and process input information. Image or text information, after being encoded and mapped, can be converted into feature vectors with a predefined latent space dimension, and subsequent model calculations can be performed within this latent space.

[0034] Linear projection: The process of transforming input features from one feature space to another using a linear mapping relationship.

[0035] One-hot encoding: An encoding method that uses different positions in a vector to represent different discrete categories, where the position corresponding to the current category has a preset valid value, and the other positions have another preset value.

[0036] Layer normalization: This is a process of normalizing the feature dimensions of corresponding feature vectors in a neural network to adjust the numerical distribution of different feature components.

[0037] Multilayer Perceptron (MLP): A neural network structure consisting of one or more fully connected network layers, used to map and classify input features.

[0038] Self-attention: A model computing mechanism that calculates attention weights between different input features based on the degree of correlation between input features, and aggregates information from the input features based on the attention weights.

[0039] The Softmax function, also known as the normalization exponential function, is a function that converts multiple input values ​​into normalized weights, ensuring that the converted weights are within a preset range and form a relative weight relationship.

[0040] To enable those skilled in the art to better understand the technical solution of this application, the technical problem addressed by this application will be further explained below in conjunction with existing security protection methods for cross-modal intelligent agents.

[0041] As large language models and visual language models are increasingly applied to computer operation, mobile terminal operation, and graphical user interface interaction, cross-modal agents can acquire current screen images, recognize text, icons, windows, and controls on the screen, and generate corresponding interface operations based on user input. In this type of agent, the information received by the model is no longer limited to the text directly input by the user, but also includes external information carried by screen images, such as web page content, application interfaces, information returned by third-party tools, and user-generated content. Since this information is usually converted into visual features after being processed by the visual encoder and then input into the model, information from different sources may participate in subsequent reasoning within the model in the form of the same or similar visual features.

[0042] For the behavioral security issues of intelligent agents driven by large language models and visual language models, the behavioral security control and anti-jailbreak attack technologies of related technologies mainly include: (1) a security control method based on text instruction level, which sets different permission levels for system instructions, current user instructions and third-party tools or external data, so that when information from different sources has semantic conflicts, the model responds to instructions with higher permissions first, and can further set input security fences, runtime monitoring and output action restrictions at the artificial intelligence application layer to perform security detection on the model's input, reasoning process and output behavior; (2) a multimodal security alignment method based on visual adversarial training, which adds negative samples containing deceptive text, image banners or other visual attack content during the multimodal instruction fine-tuning process, so that the model learns the corresponding visual attack features and improves the model's ability to recognize visual jailbreak content; (3) a protection method based on visual token, which desensitizes, perturbs or erases the token corresponding to the screen user interface through local encoder, and can be matched in combination with the semantics of the interface structure to reduce the impact of untrusted or sensitive information on the model.

[0043] While the aforementioned technologies can improve the security of intelligent agents to some extent, they still have limitations when applied to cross-modal intelligent agents that need to directly perceive and operate the graphical user interface of a computer or mobile terminal. For traditional text interaction, information from different sources typically has explicit identifiers such as system roles, user roles, or third-party data, thus allowing for the direct establishment of corresponding instruction permission levels. However, cross-modal intelligent agents primarily perceive the graphical user interface through screen images. Operating system native components, third-party application interfaces, web page content, and user-generated content all appear as continuous pixel data. When this pixel data is processed by a visual encoder to form a visual token, the visual token mainly carries the visual semantics of the corresponding area. The window, process, and permission source to which the original content belongs are usually not directly determined from the pixel content itself. Therefore, even if a web page displays text or an interface similar to system prompts, relying solely on visual semantics makes it difficult to reliably distinguish between real system components and simulated web page content, making it difficult to directly apply the instruction hierarchy established based on text roles to the visual space.

[0044] Meanwhile, security fences set up at the AI ​​application layer are essentially based on the semantics of the input content for classification and risk assessment. They are typically independent of the operating system's window management process and cannot directly obtain operating system-side information such as the process identity and window origin of the corresponding window based on screen pixels. For example, when a webpage embeds an interface similar to a system permission request window, although the security fence can recognize the text and visual semantics contained within, it is difficult to determine whether it is a genuine native operating system window or a simulated window drawn by the webpage based solely on the visual content. Therefore, there is a lack of direct correspondence between the application-layer semantic detection results and the actual permission source of the screen visual content.

[0045] Furthermore, security alignment methods based on visual adversarial training primarily rely on existing attack samples in the training data to learn visual attack features. When the text format, graphic style, or interface layout of the visual jailbreak content changes, further security training using corresponding samples is required. Therefore, it exhibits a certain lag in responding to constantly evolving visual attack forms, and continuous multimodal security fine-tuning of the model may also affect the model's original general visual semantic understanding capabilities. Existing visual token-level protection methods mainly achieve information protection through desensitization, perturbation, or erasure. Their focus is on reducing the information content contained in the visual token, but they cannot further establish permission attributes derived from the operating system window identity for the visual token, nor can they effectively limit the control effect of low-privilege visual content on the agent's behavioral decisions.

[0046] In summary, during the operation of a cross-modal intelligent agent's graphical user interface, the relevant technologies have the technical problem of making it difficult to accurately distinguish the credibility of visual information from different sources in a complex graphical user interface environment, which leads to untrusted visual information easily interfering with the cross-modal intelligent agent's reasoning process.

[0047] To address the aforementioned problems, this application discloses the following technical solution.

[0048] Reference Figure 1 , Figure 1 A flowchart of a cross-modal intelligent agent secure reasoning method based on permission identification is shown. According to a first aspect of an embodiment of this application, this application provides a cross-modal intelligent agent secure reasoning method based on permission identification. The method includes:

[0049] Step 10: Obtain the image data and window metadata corresponding to the current physical screen.

[0050] Step 20: Determine the window permission identifier of each window in the image data based on the window metadata, and label each visual block in the image data based on the window permission identifier to obtain the permission visual block.

[0051] Step 30: Encode the permission visual tiles using a visual encoder to generate visual feature vectors, and generate privileged feature vectors based on the visual feature vectors.

[0052] Step 40: Classify and predict the control intent probability by passing the privileged feature vector through the intent classification head.

[0053] Step 50: Generate a privileged attention mask matrix based on the privileged feature vector and the control intent probability, and apply permission constraints to the model inference process based on the privileged attention mask matrix to obtain a secure inference result with permission constraints.

[0054] Unlike methods that rely solely on screen images for visual semantic reasoning, this application acquires image data and window metadata of the current physical screen. It uses the window metadata to determine the window permission identifier for each window and further maps these identifiers to visual tiles. This ensures that visual tiles possess permission attributes corresponding to their information source before entering visual encoding and model reasoning. Based on this, visual feature vectors are combined with permission information to form privileged feature vectors, and a privileged attention masking matrix is ​​generated using control intent probabilities. This allows the permission attributes corresponding to the window source to be progressively passed from the image space layer to the model feature layer and attention calculation layer. Therefore, when performing cross-modal reasoning, the model no longer treats all visual information on the physical screen as equally reliable. Instead, it applies differentiated attention constraints to different visual information based on the source permission and control intent, reducing the impact of visual content in low-permission or untrusted windows on the model's reasoning process. This improves the reliability of cross-modal agent reasoning results and the security of operational decisions in complex graphical user interface environments.

[0055] In one embodiment of this application, based on the foregoing scheme, step 10 of this application, obtaining the image data and window metadata corresponding to the current physical screen, includes:

[0056] Step 110: Call the screen capture interface provided by the operating system to obtain the image data corresponding to the current physical screen.

[0057] Specifically, when a cross-modal agent triggers a screen perception or interface operation request, it invokes the operating system's graphics display service or screen mirroring service to capture an image of the current physical screen, thereby obtaining an image of the current physical screen rendered and output by the graphics processor. The image data can be full-screen RGB (a three-dimensional color space based on red, green, and blue) pixel data, maintaining the same spatial resolution as the current physical screen.

[0058] Step 120: Use the operating system's window manager to traverse the active and visible windows on the current physical screen, obtain the spatial boundary coordinates and process identity characteristics of each window, and obtain the window metadata.

[0059] Specifically, while acquiring image data, the operating system's window compositor, window manager, application framework layer interface, or corresponding accessibility interface can be invoked to traverse the active and visible windows or user interface components on the current screen and obtain the spatial boundary coordinates of each window. Thus, through two data channels—image data acquisition and window metadata acquisition—the actual pixel content displayed on the current screen and the corresponding window's operating system-level identity information are obtained, respectively. Both use the same physical screen coordinate system, enabling the establishment of a spatial correspondence between window metadata and image data in subsequent processing.

[0060] The spatial boundary coordinates are used to characterize the spatial extent of the corresponding window in the current physical screen coordinate system, for example, by using a window rectangle (Rect). For each traversed window, the process identity characteristics bound to that window are also obtained. These process identity characteristics may include at least one of the following: process identifier (PID), user identifier (UID), application package name, and control type attribute.

[0061] This application acquires image data of the current physical screen by calling the screen capture interface provided by the operating system, and traverses the active and visible windows through the window manager to synchronously obtain the spatial boundary coordinates and process identity characteristics of each window. This ensures that both the visual image data and the window metadata used for permission determination originate from the actual operating system environment currently being displayed and running. The spatial boundary coordinates provide a locational basis for mapping window permissions to the image space, while the process identity characteristics provide a source basis for determining the security trust domain and permission level to which the window belongs. This allows subsequent permission labeling to be based on the actual window state of the operating system rather than semantic guesses of the screen content by the model. Therefore, it can improve the spatial and source consistency between window permission identifiers and the current screen visual content, reduce the uncertainty caused by inferring the credibility of information solely based on image content, and provide a reliable raw data foundation for subsequent visual tile permission labeling.

[0062] In one embodiment of this application, based on the aforementioned scheme, in step 20 of this application, the window permission identifier of each window in the image data is determined according to the window metadata, and each visual patch in the image data is labeled based on the window permission identifier to obtain the permission visual patch, which includes:

[0063] Step 210: Match the process identity characteristics with the preset security trust domain, and determine the window permission identifier corresponding to the window's permission level based on the matching result.

[0064] Specifically, the system pre-establishes a security trust domain, which records the correspondence between different process identity characteristics and corresponding permission levels. After obtaining the process identity characteristics corresponding to each window, the process identity characteristics are matched with the identity information in the security trust domain to determine the permission level of the corresponding window. For example, native operating system components can be set to a high permission level; third-party application windows, web page areas, or user-generated content can be set to a low permission level. Based on the determined permission level, a window permission identifier is assigned to the corresponding window. The window permission identifier can be represented by a preset permission code, allowing different windows to be distinguished by their corresponding permission codes. For example, the high permission code corresponding to native operating system components is 0x01, and the low permission code corresponding to third-party application windows, web page areas, or user-generated content is 0x09.

[0065] Step 220: Map the window permission identifier to the pixel region in the image data corresponding to the spatial position of the window based on the spatial boundary coordinates to obtain the security level domain.

[0066] Specifically, an identity mask matrix with the same spatial dimensions as the image data can be established in memory. Based on the spatial boundary coordinates of each window, the window permission identifier is written into the corresponding spatial region of the identity mask matrix to obtain the security level domain. For example, for a rectangular region defined by the spatial boundary coordinates, the window permission identifier of the corresponding window is filled into the same coordinate interval in the identity mask matrix. After mapping each window, each pixel position in the identity mask matrix has a corresponding permission identifier, thus forming multiple security level domains aligned with the physical screen image space.

[0067] Therefore, the security level domain and the image data use the same spatial coordinate system, so that the pixel content in the screen image can be associated with the corresponding window permission identifier through its spatial coordinates.

[0068] Step 230: Divide the image data into multiple visual blocks according to the preset standard pixel range.

[0069] Specifically, based on the input tile size corresponding to the visual encoder, the image data is divided according to a fixed pixel size. For example, a complete screen image can be sequentially divided into multiple rectangular visual tiles according to a preset length and width, giving each visual tile a defined spatial boundary coordinate. Through this division, a sequence of visual tiles composed of multiple visual tiles can be obtained. Each visual tile not only contains the pixel content of the corresponding region but also contains tile boundary coordinates that can be spatially compared with the security level domain.

[0070] For example, a screenshot with data dimensions H×W×C can be divided into a discrete sequence of visual tiles X=[x1,x2,...,x] according to a fixed size (e.g., a 16×16 pixel interval). n ], where each visual tile x i The dimension is 16×16×C, and the total number of visual tiles n=HW / (16×16).

[0071] Step 240: Determine the target security level domain that spatially overlaps with each visual patch through spatial geometric indexing, and calculate the overlap area between each visual patch and the corresponding security level domain.

[0072] Specifically, for any visual tile, a spatial query range is established based on its spatial boundary coordinates, and a security level domain that overlaps with this spatial query range is retrieved using a spatial geometric index. An R-Tree spatial index can be used. For each retrieved security level domain, the pixel overlap area between it and the current visual tile is calculated, thereby determining the spatial range occupied by the security level domain corresponding to different permission levels within the current visual tile.

[0073] Step 250: Determine the block permission identifier of each visual block based on the overlapping area to obtain the permission visual blocks.

[0074] This application matches the process identity characteristics of a window with a preset security trust domain, converting the window source at the operating system level into a window permission identifier. It then maps the window permission identifier to the corresponding pixel region in the screen image based on the window's spatial boundary coordinates, thereby establishing a permission correspondence between the operating system window and the image's visual region. Furthermore, the image is divided into visual tiles, and the tile permission identifier for each visual tile is determined based on the spatial overlap between the visual tile and the security level domain. This ensures that permission information originally applied to window-level objects can be accurately transmitted to the tile-level input unit processed by the visual encoder. Therefore, each visual tile processed by the visual encoder carries permission attributes corresponding to its actual window source, reducing misjudgments caused by judging credibility solely based on the visual content itself. This provides a spatially corresponding permission foundation for subsequent privilege feature construction and permission constraint reasoning.

[0075] In one embodiment of this application, based on the aforementioned scheme, in step 250 of this application, the block permission identifier of each visual block is determined according to the overlapping area, and the permissioned visual blocks include:

[0076] Step 251: Calculate the area ratio of each target security level domain in the corresponding visual tile based on the overlapping area.

[0077] Specifically, for each target security level domain corresponding to a visual tile, the overlapping area between the target security level domain and the visual tile is divided by the total area of ​​the visual tile to obtain the area ratio of the target security level domain in the visual tile.

[0078] Step 252: When the area ratio corresponding to any target security level domain is greater than or equal to the preset subject attribution threshold, the window permission identifier corresponding to the target security level domain is marked as the block permission identifier of the visual block, and the permission visual block is obtained.

[0079] The subject attribution threshold is used to determine whether a visual tile is primarily located within a certain security level domain. For example, the subject attribution threshold can be set to 80%. Specifically, when the area proportion of a certain security level domain in a visual tile reaches or exceeds the subject attribution threshold, it indicates that the main pixel content of the visual tile originates from that security level domain. Therefore, the window permission identifier corresponding to that security level domain is used as the tile permission identifier for the current visual tile.

[0080] Step 253: When the area ratio corresponding to any target security level domain is less than the subject attribution threshold, the window permission identifier corresponding to the target security level domain with the lowest permission level is marked as the block permission identifier of the visual block, and the permission visual block is obtained.

[0081] Specifically, when a visual tile spans two or more different security level domains, and there is no single security level domain whose area proportion reaches the subject attribution threshold, it indicates that the visual tile belongs to the boundary tile between different permission areas. In this case, the target security level domain with the lowest permission level is determined from all target security level domains overlapping with the current visual tile, and its corresponding window permission identifier is used as the tile permission identifier of the current visual tile.

[0082] This application calculates the area proportion of each target security level domain in a visual tile and uses a subject attribution threshold to determine whether the visual tile has a clear primary window source. When a target security level domain with an area proportion reaching the subject attribution threshold exists, the tile permission identifier is determined based on the security level domain occupying the main area of ​​the visual tile, ensuring that the tile permission reflects the source permission of its main visual content. When a visual tile spans multiple security level domains and does not have a clear primary area, the window permission identifier with the lowest permission level is used to label the visual tile, causing mixed visual tiles that cross window boundaries to be processed with lower permissions. This avoids excessive reduction of visual tile permissions due to minor boundary overlaps and prevents low-permission content from gaining higher permissions by leveraging high-permission content when high-permission and low-permission areas coexist in the same visual tile. It achieves robust permission labeling for window boundary areas and multi-window overlapping areas, improving the reliability of the match between tile permission identifiers and the actual source of visual information.

[0083] In one embodiment of this application, based on the foregoing scheme, step 30 of this application, encoding the permission visual patch using a visual encoder to generate a visual feature vector, and generating a privileged feature vector based on the visual feature vector, includes:

[0084] Step 310: Input the permission visual tiles into the visual encoder for encoding to generate visual feature vectors corresponding to each permission visual tile.

[0085] Specifically, the permission visual patches obtained in step 20 are input into the visual encoder in a sequence. The visual encoder extracts visual semantic features from the pixel content of each permission visual patch and transforms them through linear projection to the latent space dimension corresponding to the cross-modal large model, thereby forming the visual feature vector corresponding to each permission visual patch. The visual feature vector is mainly used to characterize the image content and visual semantics contained in the corresponding permission visual patch, while the patch permission identifier is used to characterize the source of the operating system permission corresponding to the visual content.

[0086] For example, a sequence of permission-enabled visual tiles is linearly projected through a visual encoder to generate a standard visual feature vector stream V=[v1,v2,...,v...]. n ].

[0087] Step 320: Map the block permission identifier corresponding to each permission visual block to a permission feature vector with the same dimension as the visual feature vector through a preset privilege mapping matrix.

[0088] Specifically, the privilege mapping matrix is ​​used to establish the mapping relationship between discrete tile permission identifiers and the model latent space. For the i-th permissioned visual tile, its tile permission identifier is mapped through the privilege mapping matrix to obtain a permission feature vector with the same dimension as the visual feature vector.

[0089] In one implementation, the tile permission identifier can be represented using one-hot encoding. One dimension of the privilege mapping matrix corresponds to the preset number of permission levels, and the other dimension corresponds to the latent space dimension of the large model. By mapping the tile permission identifier to the privilege mapping matrix, a high-dimensional permission representation of the corresponding permission level can be selected or combined.

[0090] Step 330: Perform feature fusion between the visual feature vectors corresponding to each permission visual patch and the permission feature vectors to obtain the privileged feature vectors corresponding to each permission visual patch.

[0091] Specifically, for the i-th privileged visual patch, its privileged feature vector can be determined according to the following formula:

[0092] ;

[0093] In the formula, Represents the privileged feature vector of the i-th privileged visual patch; Represents the visual feature vector of the i-th permission visual patch; W represents the dynamic scaling factor. proj P represents the preset privileged mapping matrix; (i) W represents the tile permission identifier for the i-th permissioned visual tile; proj ·P (i) Let represent the permission feature vector of the i-th permission visual patch; LayerNorm represents the layer normalization function.

[0094] This application uses a pre-defined privilege mapping matrix to convert discrete tile permission identifiers into permission feature vectors with the same dimension as the visual feature vectors. It then fuses these permission feature vectors with the corresponding visual feature vectors, transforming the discrete permission information, originally belonging to the operating system level, into a continuous feature representation capable of participating in subsequent vector operations of the model. Through this processing, permission information is no longer treated as an external label independent of visual features, but is directly embedded in the feature representation of each visual tile. This allows the model's subsequent intent classification and attention calculation to simultaneously perceive visual semantic information and the corresponding information source permissions. This enables the transfer of permission attributes from the operating system metadata space to the model's latent feature space, improving the directness of permission information's participation in cross-modal reasoning and providing a unified feature foundation for subsequent implementation of differentiated attention constraints based on permissions.

[0095] In one embodiment of this application, based on the aforementioned scheme, in step 40, the privileged feature vector is classified and predicted using an intent classification head to obtain the probability of control intent. The intent classification head can employ one or more perceptron layers to form a classification network, using the privileged feature vector as classification input to predict the probability that the visual information contains controlling or aggressive command semantics.

[0096] For visual tiles categorized as having low privilege levels, the intent classification head reads the visual semantic features from their corresponding privileged feature vectors, obtains continuous probability values ​​through classification mapping, and uses these probability values ​​as the control intent probability. A higher control intent probability indicates a higher probability that the corresponding visual content has controlling or aggressive command semantics; a lower control intent probability indicates that the corresponding visual content is more likely to contain ordinary, objective information.

[0097] During system initialization or alignment fine-tuning, the backbone network parameters of the multimodal large model can be frozen, and the privileged mapping matrix and intent classification head can be trained using training data containing both regular objective content and malicious jailbreak commands. Specifically, the parameters of the privileged mapping matrix and intent classification head can be jointly updated using the cross-entropy loss function, while keeping the original multimodal large model backbone network parameters unchanged.

[0098] In one embodiment of this application, based on the aforementioned scheme, in step 50 of this application, a privileged attention masking matrix is ​​generated according to the privileged feature vector and the control intent probability.

[0099] Specifically, before the model enters the self-attention calculation, a privileged attention mask matrix corresponding to the current feature sequence length is dynamically generated based on the privilege level corresponding to each privileged feature vector and the control intent probability output by the intent classification head. The element in the i-th row and j-th column of the privileged attention mask matrix represents the attention penalty applied when the i-th privileged feature vector is read when the j-th privileged feature vector is read. The attention penalty is not a fixed setting, but is determined jointly based on the privilege hierarchy relationship between the two privileged feature vectors and the corresponding control intent probability.

[0100] In one implementation, the element values ​​in the privileged attention masking matrix can be determined according to the following formula:

[0101] ;

[0102] In the formula, This represents the value of the element in the i-th row and j-th column of the privileged attention masking matrix; Privilege(x) represents the privilege level corresponding to the x-th privileged feature vector. The higher the privilege level, the smaller the value of Privilege(x). This represents the probability of control intent corresponding to the i-th privileged feature vector; This indicates a preset threshold for tolerance of intent. This represents the attention decay constant. It should be noted that the magnitude and direction of the permission level values ​​can be set according to the permission encoding method used by the system, as long as there is a definite correspondence between different permission identifiers and permission levels. The control intent probability output by the intent classification header enables the privileged attention masking matrix to have three-state routing capabilities: high-privilege content is completely allowed, malicious low-privilege content is completely blocked, and benign low-privilege content is moderately decayed.

[0103] This application constructs a privileged attention masking matrix by combining the privileged feature vector with the corresponding privilege level and control intent probability. It then uses an intent tolerance threshold to distinguish different control intent states and determines the corresponding attention penalty level through an attention decay constant. This allows the attention masking to go beyond simply masking based on the privilege level of the visual content; it comprehensively considers both the source privilege of the visual information and the likelihood that the visual information will trigger control behavior. Therefore, when low-privilege visual information has a high control intent, stronger attentional inhibition can be applied during the attention calculation stage. Conversely, for visual information with low control intent or within the permissible range, unnecessary strong constraints are avoided. This allows the model to form differentiated attentional restrictions for visual information with different risk levels, reducing the possibility of low-privilege visual content inducing the agent to produce unintended control behavior, while also minimizing excessive interference with the normal use of visual semantic information.

[0104] In one embodiment of this application, based on the aforementioned scheme, in step 50 of this application, the privileged attention masking matrix is ​​used to impose permission constraints on the model inference process to obtain a permission-constrained secure inference result, including:

[0105] Step 510: Perform self-attention calculation on the privileged feature vector to obtain the initial attention value, and apply a penalty constraint to the initial attention value based on the privileged attention mask matrix to obtain the privileged attention value.

[0106] Specifically, the privileged feature vector stream, consisting of multiple privileged feature vectors, is input into the self-attention layer of the cross-modal large model. Based on the corresponding linear mapping parameters in the model, the privileged feature vectors are converted into query vectors, key vectors, and value vectors, respectively. The query vectors form a query matrix Q, the key vectors form a key matrix K, and the value vectors form a value matrix V.

[0107] First, calculate the initial attention values ​​between different privileged feature vectors based on the query matrix Q and the key matrix K, i.e.:

[0108] ;

[0109] Where S represents the initial attention value; Q represents the query matrix; K represents the key matrix; d k This represents the dimension of the key vector.

[0110] After obtaining the initial attention value, the privileged attention mask matrix generated above is directly applied to the initial attention value, and the penalty constraint is completed before the normalized exponential function is calculated, thus obtaining the privileged attention value:

[0111] ;

[0112] In the formula, Q represents the privileged attention value; K represents the query matrix; V represents the key matrix; d represents the value matrix; k Indicates the dimension of the key vector; M represents the initial attention value; privilege represents the privileged attention masking matrix; softmax(·) represents the normalized exponential function.

[0113] The above formula enables asymmetric route interception targeting different semantic intents:

[0114] To suppress malicious jailbreak commands, when a high-privilege system Token i reads a low-privilege webpage Token j during model inference, and the low-privilege webpage Token j contains malicious jailbreak commands, causing the probability of its corresponding control intent to reach a preset intent tolerance threshold, the corresponding element in the privileged attention masking matrix is ​​masked. Set to a value approaching positive infinity. Therefore, when performing self-attention calculations, through the aforementioned element... A penalty is applied to the initial attention value between the high-privilege system Token i and the low-privilege webpage Token j. After calculation using a normalized exponential function, the attention weight corresponding to the low-privilege webpage Token j is made close to zero, thereby suppressing the transmission of malicious jailbreak instructions from the webpage to the high-privilege system Token and reducing the impact of the malicious jailbreak instructions on the subsequent inference results of the model.

[0115] Regarding the retention of useful webpage information, when a high-privilege system Token i reads a low-privilege webpage Token j during model inference, and the low-privilege webpage Token j contains bill numbers, waybill numbers, or other objective webpage text, such that the probability of its corresponding control intent is lower than a preset intent tolerance threshold, the corresponding element in the privileged attention masking matrix is... The attention decay value γ is set as a finite attention decay value. Therefore, during self-attention calculation, the attention weight of the corresponding low-privilege webpage token is reduced by this finite attention decay value, but the semantic information of the webpage carried by its value vector is retained and continued to be transmitted to subsequent networks of the model. This allows the model to still read the bill number, waybill number, or other objective webpage text, and continue to perform tasks such as text summarization, data comparison, and information extraction based on the objective webpage information.

[0116] Because the privileged attention masking matrix is ​​applied to the initial attention value before Softmax calculation, features from different privilege sources can have their normalized attention weights altered based on their corresponding masking values. For example, when the masking element corresponding to a feature requiring restriction is set to a large penalty value, the initial attention value corresponding to that element is significantly reduced before Softmax is executed, and its normalized attention weight decreases accordingly. When the masking element approaches positive infinity, the corresponding attention weight approaches zero, thereby blocking the corresponding feature during the current attention information transmission process.

[0117] For low-authority objective information whose control intent probability does not reach the intent tolerance threshold, a finite attention decay constant can be used as a penalty, allowing this low-authority information to still participate in subsequent feature propagation through the value vector V, but its attention effect is limited. Therefore, while limiting the influence of low-authority control information on model behavior, it is not necessary to completely block ordinary objective information in low-authority regions.

[0118] Step 520: Perform model inference based on privileged attention values ​​to obtain secure inference results constrained by permissions.

[0119] Since the attention propagation path has already been constrained based on the permission level and control intent probability of visual information during the model's self-attention calculation, the latent space features upon which subsequent model inference is based have already undergone permission constraints. For low-permission control information that meets the constraints, its information transmission to the corresponding features is suppressed; for objective visual information with low permission but no obvious control intent, limited information transmission capability is retained. Therefore, this application does not perform security filtering on the results after the model output is generated, but rather maps the permission attributes from the operating system window metadata to visual tiles during the process of visual information entering the model inference, and further embeds them into the model's latent space. The privileged attention masking matrix directly acts on the self-attention calculation process, allowing permission constraints to participate in the internal feature propagation of the model, thereby obtaining a secure inference result constrained by permissions.

[0120] This application first calculates the initial attention value by performing self-attention on the privileged feature vector, and then applies a penalty constraint to the initial attention value using a privileged attention masking matrix. This allows access control to directly affect the strength of attention associations between different features within the model, rather than simply deleting visual content before model input or filtering inference results after model output. By changing the relative contributions of different privileged features in the attention calculation process, the propagation of low-privilege, high-risk visual information during the information aggregation process within the model is suppressed, while visual information that meets access requirements can maintain normal information interaction. Therefore, it can reduce the impact of untrusted visual information on contextual feature aggregation and subsequent decisions during the model's inference process, enabling an endogenous integration of access constraints and the model's inference process, and improving the direct impact of security constraints on the inference process.

[0121] The specific embodiments of this application have been described above. The application embodiments of this application are further described below to illustrate the method of this application. (Refer to...) Figure 2 The diagram shows the data interaction of a cross-modal intelligent agent secure reasoning method based on permission labeling according to this application.

[0122] In one application example, suppose a user inputs an instruction to a cross-modal agent: "Please summarize the email currently open on the screen." The corresponding email window is displayed on the current physical screen. In addition to the normal email content, the email body also contains low-visibility text hidden within the email body. This low-visibility text includes control content that instructs the agent to ignore the user's original instruction and send the local address book to a specified external address.

[0123] First, when the cross-modal agent perceives the current physical screen, it calls the screen capture interface provided by the operating system to obtain the image data corresponding to the current physical screen. At the same time, it uses the window manager of the operating system to traverse the windows in the current physical screen that are active and visible to obtain the spatial boundary coordinates and process identity characteristics of the email window to form window metadata.

[0124] Subsequently, the process identity features corresponding to the email window are matched with the preset security trust domain. Based on the matching result, the window permission identifier corresponding to the email window is determined. Then, based on the spatial boundary coordinates of the email window, the window permission identifier is mapped to the pixel area in the image data corresponding to the email window to form the corresponding security level domain.

[0125] The image data corresponding to the current physical screen is divided into multiple visual tiles according to a preset standard pixel range. A spatial geometric index is used to determine the security level domains that spatially overlap with each visual tile. The tile permission identifier for each visual tile is determined based on the overlap area between the visual tile and its corresponding security level domain. For visual tiles located in the email body area, they are designated as low-permission visual tiles according to the permission level corresponding to the email window.

[0126] Subsequently, the permission-marked visual blocks are input into the visual encoder, which encodes each permission visual block to obtain the corresponding visual feature vector. At the same time, the block permission identifier of each permission visual block is mapped to a permission feature vector with the same dimension as the visual feature vector through a preset privilege mapping matrix. The visual feature vector and the corresponding permission feature vector are then fused to obtain a privilege feature vector carrying the corresponding permission attributes.

[0127] For privileged feature vectors corresponding to the email body, further classification and prediction are performed using an intent classification head. For privileged feature vectors corresponding to normal email body, the intent classification head obtains a low probability of control intent; for privileged feature vectors containing instructions to ignore previous ones or requests to send the local address book to external addresses, the intent classification head identifies them as visual information with strong control semantics and obtains a probability of control intent that reaches a preset intent tolerance threshold.

[0128] When the model performs self-attention calculation, a privileged attention mask matrix is ​​generated based on the privilege level and control intent probability corresponding to each privileged feature vector. When a high-privilege feature in the model reads a low-privilege privileged feature vector in the email body whose control intent probability reaches a preset intent tolerance threshold, a large attention penalty is applied to the corresponding initial attention value, causing the attention weight of the corresponding privileged feature vector after normalization exponential function calculation to approach zero.

[0129] Therefore, although the control content with prompt-injection characteristics in the email body still exists in the image data of the current physical screen, its information transmission to the model's subsequent inference process is suppressed by the privileged attention masking matrix; while the normal objective content in the email body can still participate in the subsequent model inference. The model ultimately generates the corresponding secure inference result based on the user's input email summary command and the retained normal email content, thus completing the summary of the current email content, without performing operations such as sending local contacts based on low-privilege control content in the email body.

[0130] In another application example, suppose a user inputs an instruction to a cross-modal agent: "Please fill in the tracking number from this webpage into Excel." The current physical screen simultaneously displays a webpage window containing the tracking information and an Excel window for data entry. The webpage displays objective data such as the tracking number X.

[0131] First, when the cross-modal agent acquires the image data corresponding to the current physical screen, it simultaneously acquires the spatial boundary coordinates and process identity characteristics of the web page window and the Excel window through the window manager of the operating system. Based on the matching results between the process identity characteristics of each window and the preset security trust domain, it determines the window permission identifier corresponding to each window.

[0132] Specifically, based on the process identity characteristics corresponding to the webpage window, the webpage window is determined to have a relatively low permission level, and the corresponding window permission identifier is mapped to the corresponding pixel area in the image data according to the spatial boundary coordinates of the webpage window, so as to form the security level domain corresponding to the webpage window.

[0133] Subsequently, the image data corresponding to the current physical screen is divided into multiple visual tiles, and the tile permission identifier corresponding to each visual tile is determined based on the spatial overlap relationship between each visual tile and the security level domain. Among them, the visual tile corresponding to the webpage area displaying the waybill number X is marked as a low-permission visual tile.

[0134] The aforementioned permission visual tiles are encoded using a visual encoder to obtain corresponding visual feature vectors. Based on their tile permission identifiers, corresponding permission feature vectors are generated using a privilege mapping matrix. Finally, the visual feature vectors and permission feature vectors are fused to obtain privilege feature vectors corresponding to the webpage content.

[0135] Furthermore, the privileged feature vector corresponding to the webpage region is input into the intent classification head for classification prediction. Since the waybill number X is only used to represent objective business data on the webpage and does not contain control semantics that require the model to change the current task, perform additional operations, or override the user's original instructions, the probability of control intent output by the intent classification head is lower than the preset intent tolerance threshold.

[0136] When the model performs self-attention calculation, based on the low privilege level of the privileged feature vector corresponding to the webpage region and the control intent probability below the intent tolerance threshold, a limited attention decay value is set for the corresponding feature in the privileged attention masking matrix, instead of using a strong penalty value that makes the attention weight approach zero.

[0137] Therefore, after penalizing the initial attention values ​​using the privileged attention masking matrix, although the attention weights of the privileged feature vectors corresponding to the webpage regions are attenuated to some extent, the objective data of the tracking number X carried by its value vector can still be passed to subsequent network layers of the model and participate in the inference process of the current user operation task. The model continues to perform model inference based on the privileged attention values ​​after permission constraints, identifies and extracts the tracking number from the webpage region, and fills the tracking number into the corresponding position in Excel according to the user's input operation instructions.

[0138] Therefore, for objective data originating from low-privilege web page regions but lacking control intent, this application does not directly remove it from the model inference context. Instead, it preserves its ability to participate in inference as task data through limited attention decay. At the same time, since the web page information itself corresponds to a low permission level, it cannot rely solely on the web page content to obtain the control ability to change high-privilege task instructions, thereby achieving decoupling between data reading and instruction permissions.

[0139] The first aspect of this application has been described above. The second aspect will be described below.

[0140] According to a second aspect of the embodiments of this application, reference can be made to Figure 3 This document illustrates the architecture of a cross-modal intelligent agent secure reasoning system based on permission labeling, according to an embodiment of this application. This application provides a cross-modal intelligent agent secure reasoning system 100 based on permission labeling, including a memory 101, a processor 102, and a computer program stored in the memory 101 and running on the processor 102. When the processor 102 executes the computer program, it implements the various processing steps described in the above-described embodiment of a cross-modal intelligent agent secure reasoning method based on permission labeling. Technical content identical to that in the aforementioned method embodiment will not be repeated in the system embodiment.

[0141] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0143] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program instructions, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0145] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cross-modal intelligent agent secure reasoning method based on permission identification, characterized in that, include: Retrieve the image data and window metadata corresponding to the current physical screen; The window permission identifier of each window in the image data is determined based on the window metadata, and the visual blocks in the image data are labeled based on the window permission identifier to obtain permission visual blocks; The permission visual patch is encoded using a visual encoder to generate a visual feature vector, and a privileged feature vector is generated based on the visual feature vector. The privileged feature vector is classified and predicted using the intent classification head to obtain the control intent probability; A privileged attention masking matrix is ​​generated based on the privileged feature vector and the control intent probability. The privileged attention masking matrix is ​​then used to impose permission constraints on the model inference process, resulting in a secure inference result with permission constraints.

2. The method for secure cross-modal agent reasoning based on permission identification according to claim 1, characterized in that, The window metadata includes the spatial boundary coordinates of each window and process identity characteristics; Based on the window metadata, the window permission identifier of each window in the image data is determined, and based on the window permission identifier, each visual tile in the image data is labeled to obtain permission visual tiles including: The process identity features are matched with a preset security trust domain, and a window permission identifier corresponding to the permission level of the window is determined based on the matching result. Based on the spatial boundary coordinates, the window permission identifier is mapped to the pixel region in the image data corresponding to the spatial position of the window to obtain the security level domain; The image data is divided into multiple visual blocks according to a preset standard pixel range; The target security level domains that spatially overlap with each of the visual patches are determined by spatial geometric indexing, and the overlap area between each visual patch and the corresponding security level domain is calculated. Based on the overlapping area, the block permission identifier of each visual block is determined, and the permissioned visual blocks are obtained.

3. The method for secure cross-modal agent reasoning based on permission marking according to claim 2, characterized in that, Based on the overlapping area, the block permission identifier of each visual block is determined, resulting in permissioned visual blocks including: Calculate the area ratio of each target security level domain in the corresponding visual tile based on the overlapping area; When the area ratio corresponding to any target security level domain is greater than or equal to the preset subject attribution threshold, the window permission identifier corresponding to the target security level domain is marked as the block permission identifier of the visual block, and a permission visual block is obtained. When the area ratio corresponding to any target security level domain is less than the subject attribution threshold, the window permission identifier corresponding to the target security level domain with the lowest permission level is marked as the block permission identifier of the visual block, thus obtaining the permission visual block.

4. The method for secure cross-modal agent reasoning based on permission marking according to claim 3, characterized in that, Generating privileged feature vectors based on the aforementioned visual feature vectors includes: The block permission identifier corresponding to each of the aforementioned permission visual blocks is mapped to a permission feature vector with the same dimension as the visual feature vector through a preset privilege mapping matrix; The visual feature vector corresponding to each permission visual block is fused with the permission feature vector to obtain the privilege feature vector corresponding to each permission visual block.

5. A cross-modal intelligent agent secure reasoning method based on permission marking according to claim 1, characterized in that, Generating a privileged attention masking matrix based on the privileged feature vector and the control intent probability includes: ； In the formula, This represents the value of the element in the i-th row and j-th column of the privileged attention masking matrix; Privilege(x) represents the privilege level corresponding to the x-th privileged feature vector. The higher the privilege level, the smaller the value of Privilege(x). This represents the probability of control intent corresponding to the i-th privileged feature vector; This indicates the preset threshold for tolerance of intent; This represents the attention decay constant.

6. The method for secure cross-modal agent reasoning based on permission identification according to claim 1, characterized in that, Based on the privileged attention masking matrix, permission constraints are applied to the model inference process to obtain permission-constrained secure inference results, including: The privileged feature vector is self-attentioned to obtain an initial attention value, and the initial attention value is penalized and constrained based on the privileged attention masking matrix to obtain the privileged attention value. Based on the privileged attention value, model inference is performed to obtain a secure inference result constrained by permissions.

7. A cross-modal intelligent agent secure reasoning method based on permission marking according to claim 6, characterized in that, The privileged feature vector is self-attentioned to obtain an initial attention value, and the initial attention value is penalized based on the privileged attention masking matrix to obtain the privileged attention value, which includes: ； In the formula, Q represents the privileged attention value; K represents the query matrix; V represents the key matrix; d represents the value matrix; k Indicates the dimension of the key vector; M represents the initial attention value; privilege represents the privileged attention masking matrix; softmax(·) represents the normalized exponential function.

8. A cross-modal intelligent agent secure reasoning method based on permission marking according to claim 4, characterized in that, The visual feature vector corresponding to each of the permission visual blocks is fused with the permission feature vector to obtain the privileged feature vector corresponding to each of the permission visual blocks, including: ； In the formula, Represents the privileged feature vector of the i-th privileged visual patch; Represents the visual feature vector of the i-th permission visual patch; W represents the dynamic scaling factor. proj P represents the preset privileged mapping matrix; (i) W represents the tile permission identifier for the i-th permissioned visual tile; proj ·P (i) represents the permission feature vector of the i-th permission visual patch; LayerNorm represents the layer normalization function.

9. A cross-modal intelligent agent secure reasoning method based on permission marking according to claim 2, characterized in that, The image data and window metadata corresponding to the current physical screen are obtained, including: Call the screen capture interface provided by the operating system to obtain the image data corresponding to the current physical screen; The operating system's window manager traverses the active and visible windows on the current physical screen, obtains the spatial boundary coordinates and process identity characteristics of each window, and thus obtains window metadata.

10. A cross-modal intelligent agent secure reasoning system based on permission identification, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements each step of the cross-modal intelligent agent secure reasoning method based on permission labeling as described in any one of claims 1 to 9.

Images