An image token screening method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image token screening method and device, electronic equipment and storage medium, the method comprising: obtaining text tokens of a to-be-processed text and a plurality of candidate image tokens of a to-be-processed image, the text tokens containing global text features for indicating a task intention; respectively calculating semantic correlation degrees of semantic features contained in each candidate image token and the task intention, and spatial matching degrees of spatial positions of each candidate image token and the task intention; based on the semantic correlation degrees and the spatial matching degrees, screening a target image token from the plurality of candidate image tokens; wherein the target image token is used to be input to a multi-modal reasoning model after being spliced with the text tokens, so that the multi-modal reasoning model performs reasoning calculation on the target image token based on the task intention. Accordingly, the present disclosure improves the answer accuracy of the multi-modal model in a complex multi-target scene, and realizes high-precision multi-modal interaction.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to an image token filtering method, apparatus, electronic device, and storage medium. Background Technology

[0002] In multimodal data processing tasks, input images are typically divided into multiple image patches and encoded as candidate image tokens, which, together with text tokens, constitute the model input. To avoid background noise interference and reduce computational complexity, candidate image tokens need to be filtered before inference. However, existing filtering techniques mainly rely on semantic similarity matching between candidate image tokens and text queries, lacking an effective response to spatial constraints within the text.

[0003] Specifically, when a query command contains explicit spatial location information (such as "apple on the table"), if there are multiple targets with the same semantics but different spatial locations in the scene (such as apples on the table and under the table at the same time), existing filtering techniques, which only focus on the consistency of content semantics, are very likely to misjudge interfering targets that do not meet the spatial conditions as highly relevant tokens. This results in a large number of irrelevant noise tokens being introduced into the model, which not only increases the inference load but also easily leads to model inference errors. Summary of the Invention

[0004] In view of this, the present disclosure provides an image token filtering method, the method comprising: Obtain a text token for the text to be processed and several candidate image tokens for the image to be processed, wherein the text token contains global text features for indicating the task intent; Calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent. Based on the semantic relevance and spatial matching degree, the target image token is filtered from the candidate image tokens; The target image token is concatenated with the text token and input into the multimodal inference model so that the multimodal inference model can perform inference calculations on the target image token based on the task intent.

[0005] Optionally, calculating the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: Using the global text features as the query vector, each candidate image token in the plurality of candidate image tokens is used as the key vector and the value vector in turn. Attention weights are calculated using the cross attention mechanism to obtain the attention weight of the query vector on each candidate image token in the plurality of candidate image tokens. Multiplying the attention weight by the value vector of the candidate image token yields the semantic relevance between the candidate image token and the task intent.

[0006] Optionally, the plurality of candidate image tokens maintain the spatial structure of the image to be processed, and the calculation of the spatial matching degree between the spatial location of each candidate image token and the task intent includes: Pooling is performed on the candidate image tokens to obtain global image features that characterize the overall semantics of the image; The global text features and the global image features are fused to obtain multimodal global fusion features; The multimodal global fusion features are input into a pre-trained spatial matching degree prediction network so that the output of the spatial matching degree prediction network is consistent with the spatial structure dimension of the image to be processed. Each element value in the spatial probability mask represents the spatial matching degree between the candidate image token at the corresponding spatial location and the task intent.

[0007] Optionally, training the spatial matching degree prediction network includes: Based on the inference loss output by the multimodal inference model and the regularization loss of the spatial matching degree prediction network, a supervision signal for the spatial matching degree prediction network is constructed. Based on the gradient of the supervision signal, the parameters of the spatial matching degree prediction network are updated through the backpropagation algorithm until the spatial matching degree prediction network converges.

[0008] Optionally, the step of filtering the target image token from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree includes: The semantic correlation degree between the semantic features contained in each candidate image token and the semantic intent of the task is calculated, and the element-wise product operation is performed with the spatial probability mask of the spatial location corresponding to the image token to obtain the comprehensive correlation degree of the image token. The candidate image tokens are sorted according to the overall relevance of each candidate image token; Among the sorted candidate image tokens, select the candidate image tokens with the highest overall relevance among the top preset number; The selected candidate image tokens are used as the target image tokens.

[0009] Optionally, the candidate image tokens contain deep image semantic representations. Before calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent, the method further includes: The candidate image tokens and the text tokens are concatenated and input into the large language model to obtain the candidate image tokens and the text tokens with shallow image semantic representations output by the large language model. Using text tokens containing shallow text semantic representations as query vectors, and each candidate image token containing shallow image semantic representations as key vector and value vector respectively, deep semantic extraction is performed using cross-attention mechanism to generate several candidate image tokens containing deep image semantic representations.

[0010] Optionally, calculating the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: The generated candidate image tokens containing deep image semantic representations are input into a multilayer perceptron mapping network, so that the multilayer perceptron mapping network maps the deep image semantic representations to the prediction distribution space, and obtains the prediction score corresponding to each candidate image token among the candidate image tokens containing deep image semantic representations. Among them, the prediction score corresponding to each candidate image token in the several candidate image tokens containing deep image semantic representation represents the semantic relevance between the semantic features contained in the candidate image token and the task intent.

[0011] This disclosure also provides an image token filtering device, the device comprising: The acquisition unit is used to acquire a text token of the text to be processed and a number of candidate image tokens of the image to be processed, wherein the text token contains global text features for indicating the task intent; The computing unit is used to calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent. A filtering unit is used to filter target image tokens from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree. The target image token is concatenated with the text token and input into the multimodal inference model so that the multimodal inference model can perform inference calculations on the target image token based on the task intent.

[0012] This disclosure also provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0013] This disclosure also provides a machine-readable storage medium storing machine-readable instructions that, when called and executed by a processor, implement the above-described method.

[0014] Therefore, the technical solution disclosed herein constructs an image token filtering mechanism with both semantic and spatial dimensions. This disclosure makes the implicit spatial constraints in the text explicit and transforms them into quantifiable spatial matching indicators. Combined with traditional semantic similarity indicators, it performs "content relevance" and "location compliance" discrimination on each candidate image token, ensuring that the selected target image token not only responds to the text query in terms of visual content, but also strictly follows the orientation constraints of the text instructions in terms of spatial layout.

[0015] Through the above methods, this disclosure reduces the computational complexity and memory usage in the reasoning process of multimodal models, thereby improving the processing speed of multimodal models. On the other hand, it reduces reasoning errors caused by attention distraction in multimodal models, improves the accuracy of multimodal model responses in complex multi-objective scenarios, and achieves high-precision multimodal interaction. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an exemplary embodiment of dividing a plurality of candidate image tokens; Figure 2 This is a flowchart illustrating an image token filtering method as an exemplary embodiment; Figure 3 This is a flowchart illustrating another image token filtering method as an exemplary embodiment; Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device; Figure 5 This is a block diagram illustrating an image token screening device as an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0019] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this disclosure in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this disclosure. Furthermore, a single step described in this disclosure may be broken down into multiple steps in other embodiments; and multiple steps described in this disclosure may be combined into a single step in other embodiments.

[0020] In multimodal data processing tasks, input images are typically divided into multiple image patches and encoded as candidate image tokens, which, together with text tokens, constitute the model input. To avoid background noise interference and reduce computational complexity, candidate image tokens need to be filtered before inference. However, existing filtering techniques mainly rely on semantic similarity matching between candidate image tokens and text queries, lacking an effective response to spatial constraints within the text.

[0021] Specifically, when a query command contains explicit spatial location information (such as "apple on the table"), if there are multiple targets with the same semantics but different spatial locations in the scene (such as apples on the table and under the table at the same time), existing filtering techniques, which only focus on the consistency of content semantics, are very likely to misjudge interfering targets that do not meet the spatial conditions as highly relevant tokens. This results in a large number of irrelevant noise tokens being introduced into the model, which not only increases the inference load but also easily leads to model inference errors.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an exemplary embodiment of dividing a plurality of candidate image tokens. For example... Figure 1As shown, the image to be processed is an image containing a table and an apple, and is divided into three candidate image tokens: token A, token B, and token C. Token A corresponds to "the red apple on the table," token B corresponds to "the red apple on the ground below the table," and token C corresponds to "the green apple on the table." When processing the multimodal query task "Please point out the red apple on the table," the text to be processed is the query instruction "Please point out the red apple on the table." Existing image token filtering methods based solely on semantics will assign extremely high similarity scores to both token A and token B because they both contain the complete visual feature of "red apple," leading to both being potentially selected and introducing interference to the model's inference.

[0023] In view of this, this disclosure aims to propose a technical solution for image token filtering that integrates spatial location constraints and semantic relevance.

[0024] The technical solution first obtains a text token of the text to be processed and several candidate image tokens of the image to be processed. The text token contains global text features used to indicate the task intent. Further, the semantic correlation degree between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent are calculated respectively. Finally, based on the semantic correlation degree and the spatial matching degree, the target image token is selected from the several candidate image tokens.

[0025] For example, such as Figure 1 As shown, the image to be processed is an image containing a table and an apple, and is divided into three candidate image tokens: token A, token B, and token C. Token A corresponds to "a red apple on the table," token B corresponds to "a red apple on the ground below the table," and token C corresponds to "a green apple on the table." When processing the multimodal query task "Please point out the red apple on the table," the text to be processed is the query instruction "Please point out the red apple on the table." The system first parses the core semantics "table, red apple" and the spatial constraint "on the table" from the text to be processed. For token A, its semantic features are highly matched, and its spatial position is above the detected "table" area, resulting in an extremely high spatial matching degree and the highest overall score. For token B, although its semantic features are completely consistent with "red apple," its spatial position is below the "table" area, which contradicts the spatial constraint "on the table," so it is assigned a very low spatial matching degree score, causing its overall ranking to drop significantly and it to be eliminated. For token C, although the spatial position is consistent, the color semantics do not match, so it is also filtered out. Ultimately, based on the comprehensive score, the system selected only token A as the target image token.

[0026] It should be noted that the geometrical differences shown for tokens A, B, and C in the diagram are intended to visually represent the different coverage areas of each token in the original image and are intended as illustrative representations. At the technical implementation level, all candidate image tokens undergo standardized encoding and are mapped to feature vectors of a uniform dimension. Therefore, the area sizes in the diagram only represent spatial divisions and do not reflect dimensional differences in the feature data.

[0027] Therefore, the technical solution disclosed herein constructs an image token filtering mechanism with both semantic and spatial dimensions. This disclosure makes the implicit spatial constraints in the text explicit and transforms them into quantifiable spatial matching indicators. Combined with traditional semantic similarity indicators, it performs "content relevance" and "location compliance" discrimination on each candidate image token, ensuring that the selected target image token not only responds to the text query in terms of visual content, but also strictly follows the orientation constraints of the text instructions in terms of spatial layout.

[0028] Through the above methods, this disclosure reduces the computational complexity and memory usage in the reasoning process of multimodal models, thereby improving the processing speed of multimodal models. On the other hand, it reduces reasoning errors caused by attention distraction in multimodal models, improves the accuracy of multimodal model responses in complex multi-objective scenarios, and achieves high-precision multimodal interaction.

[0029] The present disclosure will now be described through specific embodiments and in conjunction with specific application scenarios.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image token filtering method as an exemplary embodiment. The method may perform the following steps: Step 202: Obtain a text token for the text to be processed and several candidate image tokens for the image to be processed, wherein the text token contains global text features for indicating the task intent.

[0031] For example, in processing the multimodal query task "Please point out the red apple on the table," the text to be processed is the query instruction "Please point out the red apple on the table." The system uses a pre-trained text encoder to perform deep semantic encoding on the acquired text to be processed, generating a text token sequence containing semantic information such as "table," "red apple," and the locative word "above," as well as a global text feature vector indicating the task intent. This global text feature vector highly condenses the core elements of the instruction: it contains both the category semantics of the target object "red apple" and the positional constraint information of the reference object "table" and the spatial orientation "above." In addition, the system divides the acquired image to be processed (a scene image containing a table and an apple) into multiple grid regions and encodes them, obtaining three candidate image tokens, namely token A, token B, and token C. Token A corresponds to the "red apple above the table" region, token B corresponds to the "red apple on the ground below the table" region, and token C corresponds to the "green apple above the table" region.

[0032] The process of obtaining the global text feature vector is as follows: The system inserts a special classification tag ([CLS]) at the beginning of the text sequence to be processed, and uses a multi-layer self-attention mechanism to allow this tag to interact and aggregate with all other word units in the text sequence (such as "table", "red apple", and "on"). Finally, the output vector corresponding to the [CLS] tag is used as the global text feature vector to indicate the task intent. This global text feature vector simultaneously encodes the visual category attribute of "red apple" and the spatial orientation constraint of "on the table", forming an overall intent representation of "finding a specific object in a specific location". The process of obtaining candidate image tokens is as follows: The visual encoder is used to segment the input image into fixed-size image blocks, and each image block is mapped to a high-dimensional vector representation. The process of obtaining the text token sequence is as follows: The text encoder is used to segment and embed the text to be processed, and each word unit (Token) after segmentation is mapped to a corresponding local word vector. These local word vectors are arranged in the original order to form a text token sequence, which contains one or more text tokens.

[0033] Step 204: Calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent.

[0034] For example, the system first parses the core semantic object "red apple" and the spatial constraint "on the table" from the text token sequence. Semantically, the system finds that tokens A and B both contain significant visual features of a "red apple," resulting in high semantic relevance scores to the task intent (e.g., 0.95), while token C, due to its "cyan" color feature, has a lower semantic relevance score (e.g., 0.3). Spatially, the system locates the reference region of the "table" in the image to be processed. Detection shows that the spatial coordinates of tokens A and C are above this reference region, conforming to the "on the table" constraint, thus resulting in high spatial matching scores to the task intent (e.g., 0.98); while the spatial coordinates of token B are below the reference region, contradicting the constraint, resulting in a lower spatial matching score (e.g., 0.05).

[0035] The spatial matching degree calculation specifically implements a spatial logical reasoning process: the image to be processed is divided into a regular grid to generate candidate image tokens; the corresponding pixel coverage area and center point coordinates are directly determined based on the index of each token in the grid; the bounding box or three-dimensional spatial range of the reference object (such as a table) is determined through object detection or depth estimation network; directional words in the text (such as "up", "down", "left", "right") are mapped to geometric judgment rules; the relative positional relationship between the center point or coverage area of ​​each candidate image token and the spatial range of the reference object is calculated; and this geometric consistency is converted into a quantified spatial matching score. The semantic relevance calculation specifically implements a cross-modal similarity measurement process: the image encoder maps each image grid block to a high-dimensional visual feature vector, which serves as the visual vector of the candidate image token; and the global text feature vector and core object word vector output by the text encoder are mapped to a semantic space shared with the candidate image tokens; by calculating the similarity between the above global feature vector, core object word vector, and the visual vector of each candidate image token, the matching degree of the two in category attributes is quantified.

[0036] Step 206: Based on the semantic relevance and the spatial matching degree, filter the target image tokens from the candidate image tokens.

[0037] For example, the system employs a weighted fusion strategy to combine the semantic relevance and spatial matching degree of each candidate image token, generating a comprehensive score for each candidate image token. Token A, possessing both high semantic relevance (e.g., 0.95) and high spatial matching degree (e.g., 0.98), has the highest comprehensive score and is therefore selected as the target image token. Token B, despite having a relatively high semantic relevance (e.g., 0.95), is eliminated because its spatial location does not match the task intent, resulting in a spatial matching degree close to zero (e.g., 0.05). Token C, although matching the task intent in spatial location (e.g., 0.98), is eliminated because its visual features do not match, leading to a very low semantic relevance (e.g., 0.3). Ultimately, the system outputs only token A as the target image token, which is then used for inference by a multimodal large model along with the text image token sequence.

[0038] The target image token is concatenated with the text token and input into the multimodal inference model, enabling the multimodal inference model to perform inference calculations based on the task intent contained in the text token for the target image token. The selection process for candidate image tokens can be as follows: construct a weighted scoring function, linearly fuse semantic relevance and spatial matching degree according to preset weights (or dynamic attention weights) or nonlinearly map them to obtain the comprehensive confidence score of each candidate image token; then perform sorting pruning or dual threshold filtering, that is, sorting according to the comprehensive score from high to low and selecting the top preset number of high-scoring candidate image tokens, or setting a minimum semantic threshold and a minimum spatial threshold, retaining only candidate image tokens that simultaneously meet both threshold conditions as target image tokens. This disclosure does not limit this selection process.

[0039] In one embodiment shown, calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent includes: using the global text features as a query vector, and each candidate image token in the plurality of candidate image tokens as a key vector and a value vector in turn, calculating attention weights using a cross-attention mechanism to obtain the attention weight of the query vector for each candidate image token in the plurality of candidate image tokens; multiplying the attention weight by the value vector of the candidate image token to obtain the semantic relevance between the candidate image token and the task intent.

[0040] For example, for the aforementioned task "Please point out the red apple on the table", the system will use the global text feature vector containing the overall intent generated in step 202. (That is, the vector corresponding to the [CLS] token output by the text encoder) is mapped to a query matrix. Query vector after application Simultaneously, the visual feature vectors corresponding to tokens A, B, and C are concatenated into an image token matrix, and the key matrix is ​​obtained through linear transformation. Sum matrix Then, the system performs the following cross-attention operation to obtain the semantic association between each candidate image token and the task intent: Among them, activation function The output is an attention weight vector of dimension N×1, denoted as . The i-th element Indicates the first The semantic relevance score of each candidate image token relative to the task intent. For example, in this embodiment, token A corresponds to... Token B corresponds to Token C corresponds to These values ​​quantify the semantic matching degree between each candidate image token and the task intent, respectively, and are correlated with the value vector of each candidate image token. Multiplying them yields a weighted feature vector, which represents the semantic correlation between each candidate image token and the task intent required for subsequent filtering.

[0041] In one embodiment, the plurality of candidate image tokens maintain the spatial structure of the image to be processed. The calculation of the spatial matching degree between the spatial location of each candidate image token and the task intent includes: pooling the plurality of candidate image tokens to obtain global image features representing the overall semantics of the image; fusing the global text features with the global image features to obtain multimodal global fusion features; and inputting the multimodal global fusion features into a pre-trained spatial matching degree prediction network so that the spatial matching degree prediction network outputs a spatial probability mask consistent with the spatial structure dimension of the image to be processed. Each element value in the spatial probability mask represents the spatial matching degree between the candidate image token at the corresponding spatial location and the task intent.

[0042] For example, for the aforementioned task "Please point out the red apple on the table", the system first performs a global average pooling operation on all candidate image tokens (preserving the spatial structure of the image to be processed, totaling...). Each candidate image token is compressed into a single global image feature vector. Simultaneously, extract the vector corresponding to the [CLS] token output by the text encoder. As global text features, the two are concatenated to form a multimodal global fusion feature. Subsequently, this global fusion feature was... Input a lightweight spatial prediction subnetwork to generate unnormalized spatial logits vectors. Finally, the Sigmoid activation function is applied element-wise to convert it into a spatial probability mask with values ​​in the range [0,1]. Each element Indicates the first The spatial matching probability values ​​between candidate image tokens at each spatial location and the task intent are given. Where the "red apple" is located in the area slightly to the right of the image center, then the corresponding location... The value is close to 1, while the corresponding value in irrelevant areas such as table legs or background walls is close to 1. The value then approaches 0.

[0043] In this network, several candidate image tokens maintain the spatial structure of the image to be processed. Each candidate image token corresponds to a visual region in the image, and the sequence index of the candidate image tokens has a definite mapping relationship with the row and column positions of the visual regions in the image grid (e.g., arranged sequentially from left to right and top to bottom). The lightweight spatial prediction subnetwork consists of multilayer perceptrons, requiring no deep convolutional structures. It can map single-point high-dimensional features back to the original spatial grid dimension through linear transformation alone, generating an unnormalized spatial matching score vector (i.e., a spatial logits vector). Spatial probability mask. The dimension N is strictly equal to the total number of candidate image tokens, and their arrangement order corresponds one-to-one with the spatial grid of the image to be processed, ensuring that each probability value can be directly associated with a candidate image token at a specific spatial location.

[0044] In one embodiment shown, training the spatial matching degree prediction network includes: constructing a supervision signal for the spatial matching degree prediction network based on the inference loss output by the multimodal inference model and the regularization loss of the spatial matching degree prediction network; and updating the parameters of the spatial matching degree prediction network using a backpropagation algorithm based on the gradient of the supervision signal until the spatial matching degree prediction network converges.

[0045] For example, the system employs an end-to-end weakly supervised learning strategy to train the spatial matching degree prediction network, without relying on additional manually labeled bounding box data. Specifically, the system calculates the cross-entropy loss (as the inference loss) based on the difference between the output of the downstream multimodal inference task and the true answer. If the spatial matching degree prediction network causes a final inference error due to selecting candidate image tokens from incorrect image regions (e.g., choosing the apple under the table instead of the apple on the table), the gradient of this inference cross-entropy loss will be passed through a spatial probability mask. The computation path propagates back to the spatial matching degree prediction network, forcing the spatial matching degree prediction network to adjust the prediction results in the next iteration, actively suppressing the response of irrelevant regions and enhancing the matching degree of key regions, until the preset stopping conditions are met (such as the inference loss being lower than the threshold or the maximum number of iterations being reached).

[0046] In some embodiments, the system may also selectively introduce auxiliary sparsity loss by adjusting the spatial probability mask. Applying L1 regularization constraints causes the spatial matching prediction network to generate sparse activation maps (i.e., spatial probability masks) focused on key regions. (Visual representation), avoiding attention diffusion that covers the entire image to be processed.

[0047] In some embodiments, the system may also selectively introduce an auxiliary smoothness loss, through the spatial probability mask. Applying L2 regularization constraints suppresses drastic jumps in matching degree between tokens of adjacent candidate images, making the activation map generated by the spatial matching degree prediction network more smooth and continuous in spatial distribution, thereby enhancing the robustness of the spatial matching degree prediction network to local noise and preventing overfitting.

[0048] The supervision signal consists of a weighted sum of inference loss and regularization loss, reflecting a training mechanism oriented towards the performance of the main task: as long as the mask generated by the spatial matching degree prediction network can guide the main model to make correct inferences, its parameters will be optimized. Since the spatial probability mask directly affects the weighted fusion result of candidate image tokens, and thus affects the final inference output, the error signal of the main task can naturally guide the spatial matching degree prediction network to learn the spatial importance distribution related to the task. The specific function of the L1 regularization constraint is to penalize the number or sum of absolute values ​​of non-zero elements, thereby causing the values ​​of most non-critical positions in the spatial probability mask to approach zero, retaining only a few high-confidence critical positions, and improving the accuracy of spatial localization. The specific function of the L2 regularization constraint is to penalize the sum of squares of the weights, limiting the amplitude range of the values ​​at each position in the spatial probability mask, causing the matching degree prediction values ​​of adjacent spatial positions to maintain a smooth transition, and improving the stability and generalization ability of the spatial mask.

[0049] In one embodiment, the step of filtering the target image token from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree includes: performing element-wise multiplication of the semantic features contained in each candidate image token with the semantic relevance of the task intent, and the spatial probability mask of the spatial location corresponding to the image token, to obtain the comprehensive relevance of the image token; sorting the plurality of candidate image tokens according to the comprehensive relevance of each candidate image token; and selecting the candidate image tokens with the highest comprehensive relevance among the sorted candidate image tokens as the target image token.

[0050] For example, taking the multimodal reasoning task of "finding the apple on the table" as an example, the system first performs spatial constraint fusion on all candidate image tokens in the image to be processed: for each candidate image token Score its semantic relevance Spatial probability mask corresponding to its spatial location Perform element-wise multiplication to obtain the overall correlation degree. Subsequently, the system sorts all candidate image tokens from high to low based on their overall relevance, and selects the top-ranked candidate image tokens (e.g., Top-5) as the target image tokens.

[0051] Assume the image to be processed contains 10 candidate image tokens ( to The task is to "find the apples on the table," with a preset filter of the first three. The system first uses a formula... Calculate the overall correlation of each candidate image token: , (The real target: the apple on the table): High scores in both semantics and spatial aspects, with overall scores of 0.81 and 0.75 respectively; (Distractor A: Apple under the table): It has high semantic relevance but extremely low spatial probability mask, and its overall score is suppressed to 0.19; (Distractor B: Water glass on the table): High spatial probability mask, but extremely low semantic relevance, and the overall score is suppressed to 0.09; to (Background): All scores are low, with the overall score less than 0.05. Next, the system sorts the 10 candidate image tokens in descending order of their overall scores and selects the top 3 candidate image tokens. , , As the target image token.

[0052] Among them, element-level multiplication operation refers to multiplying the semantic relevance with the spatial mask probability one by one to generate a "comprehensive relevance" that contains both semantic and spatial information; while "target image tokens" refer to the top preset number of candidate image tokens with the highest comprehensive scores after sorting and filtering, which will serve as effective visual input for subsequent reasoning.

[0053] In one embodiment shown, the plurality of candidate image tokens contain deep image semantic representations. Before calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent, the method further includes: concatenating the plurality of candidate image tokens and the text token and inputting them into a large language model to obtain a plurality of candidate image tokens containing shallow image semantic representations and a text token containing shallow text semantic representations output by the large language model; using the text token containing shallow text semantic representations as a query vector, and each candidate image token among the plurality of candidate image tokens containing shallow image semantic representations as a key vector and a value vector respectively, performing deep semantic extraction using a cross-attention mechanism to generate a plurality of candidate image tokens containing deep image semantic representations.

[0054] For example, the system receives several candidate image tokens obtained from the partitioning of the image to be processed. With text tokens indicating task intent The system concatenates these two sets of tokens along the sequence dimension to form a unified input sequence, which is then fed into the large language model. .Depend on Process according to the following formula: at this time, Output the two sets of features after initial fusion: These are several candidate image tokens containing shallow image semantic representations. This is a text token containing a shallow semantic representation of the text. This stage is only achieved through... The self-attention mechanism enables global context modeling, but fine-grained image-text alignment has not yet been established. Therefore, further deep semantic extraction is needed through the cross-attention module.

[0055] Subsequently, the system calls the Cross-Attention Module (CAM) to... As a query vector, Each candidate image token is used as both a key vector and a value vector. Deep semantic extraction is performed to generate a deep image semantic representation according to the following formula: The internal computation process of the Cross-Attention Module (CAM) is defined by the following formula, employing a double-nested structure to achieve progressive semantic enhancement: In the formula, the inner layer Represents each candidate image token with a shallow image semantic representation. As a query vector, a text token containing shallow textual semantic representation. For the key and value vectors, a first cross-modal reweighting is performed to generate intermediate image features guided by textual semantics. outer layer Based on this, using the aforementioned intermediate image features as the query vector, Using key and value vectors, a second image internal feature aggregation is performed to obtain several candidate image tokens containing deep image semantic representations. This strengthens local consistency and task relevance.

[0056] Among them, "shallow semantic representation" refers to Features that have not been explicitly aligned with the text after initial fusion; "deep semantic representation" is a high-order task-related feature reweighted by the CAM module with text guidance. The cross-attention mechanism achieves dynamic semantic focusing through two attention calculations: the inner CA generates "intermediate image features," i.e., image re-representation from the text perspective; the outer CA performs a second aggregation to strengthen local consistency and instruction relevance. This double-nested structure avoids information loss in a single interaction and improves feature robustness. Finally, it outputs several candidate image tokens containing deep image semantic representation. It strengthens the semantic association with the task intent, providing a feature basis for subsequent candidate image token selection based on semantic association, and effectively avoids selection bias caused by insufficient feature expression.

[0057] In one embodiment shown, calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent includes: inputting a plurality of candidate image tokens containing deep image semantic representations into a multilayer perceptron mapping network, so that the multilayer perceptron mapping network maps the deep image semantic representations to a prediction distribution space, thereby obtaining a prediction score corresponding to each candidate image token among the plurality of candidate image tokens containing deep image semantic representations; wherein, the prediction score corresponding to each candidate image token among the plurality of candidate image tokens containing deep image semantic representations represents the semantic relevance between the semantic features contained in the candidate image token and the task intent.

[0058] For example, the system will generate several candidate image tokens containing deep image semantic representations. The input multilayer perceptron mapping network is shown in the following equation: A multilayer perceptron mapping network contains at least one fully connected layer and an activation function, which represents the semantics of the input deep image. Linear transformation and mapping to The space outputs the raw prediction score representing the degree to which each candidate image token matches the task intent (i.e., Subsequently, the predicted score is normalized by performing a Sigmoid function or a Softmax function to convert it into a retained probability value in the range [0,1]. This retained probability value quantifies the semantic correlation between the semantic features contained in the candidate image token and the task intent.

[0059] in, The (log-odds) space refers to the unnormalized domain of the original predicted scores output by the neural network. The numerical value reflects the semantic correlation between the candidate image token and the task intent, and it needs to be normalized to convert it into probability values. "Deep image semantic representation" refers to the high-order image features reweighted through a two-layer cross-attention mechanism, which integrates the guiding information of the textual task intent while preserving the consistency of the local structure within the image. The multilayer perceptron mapping network is a lightweight mapping module composed of stacked fully connected layers, used to compress high-dimensional deep image semantic representations into low-dimensional ones. (Log-odds) value, while learning the feature mapping patterns related to the task.

[0060] The following combination Figure 3 This document provides a complete flowchart of the image token filtering method disclosed herein. Please refer to [link / reference]. Figure 3 , Figure 3 This is a flowchart illustrating another image token filtering method as an exemplary embodiment.

[0061] like Figure 3 As shown, the system first obtains the text token of the text to be processed. (The yellow square in the image) and several candidate image tokens of the image to be processed. (Blue square in the image). The text token contains global text features indicating task intent. Both are first input into a large language model ( The system performs initial interaction and outputs several candidate image tokens containing shallow image semantic representations. and text tokens containing shallow textual semantic representations Then, each candidate image token containing a shallow image semantic representation is used. The text tokens, which are used sequentially as query vectors (q), contain shallow textual semantic representations. Using the key vector (k) and value vector (v) as the key vector and value vector respectively, the task intent is injected into the visual features through a cross-attention mechanism to generate intermediate image features guided by textual semantics. ; using the features of this intermediate image Each candidate image token, containing shallow image semantic representations, serves as the query vector (q). Using these as key vectors (k) and value vectors (v) respectively, secondary aggregation and self-enhancement of internal image features are performed to generate several candidate image tokens containing deep image semantic representations. .

[0062] Subsequently, There are two subsequent paths, one is via The mapping network is projected onto the prediction distribution space (i.e. (space), generating tokens for each candidate image. The corresponding predicted score (i.e.) (value), that is, each candidate image token Semantic relevance relative to task intent; subsequently, it can be determined based on each candidate image token. of Sort the values ​​and select. The candidate image tokens with values ​​within the first preset number are used as the filtered image tokens. The image tokens filtered by this path have a high semantic relevance to the task intent.

[0063] Another approach is to integrate a "task-logic-based spatial region selection module," which combines various candidate image tokens. The semantic relevance between the contained semantic features and the task intent, and the spatial matching degree between their spatial locations and the task intent, are used for all candidate image tokens. Perform joint screening to select all candidate image tokens The system dynamically defines several "high-confidence visual regions," generating a visualized spatial probability heatmap (regions with higher confidence are darker in color), and extracts image tokens from the corresponding regions as filtered image tokens. This approach not only continues the advantages of deep semantic guidance but also introduces spatial location constraints. The image tokens filtered by this approach have a high degree of semantic relevance and spatial matching with the task intent.

[0064] Corresponding to the embodiments of the image token filtering method described above, this disclosure also provides an embodiment of an image token filtering device.

[0065] Please see Figure 4 , Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this disclosure can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this disclosure do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0066] Please see Figure 5 , Figure 5 This is a block diagram illustrating an image token filtering device as an exemplary embodiment. The image token filtering device 500 can be applied to, for example... Figure 4 The illustrated electronic device is used to implement the technical solution of this disclosure. The device includes: The acquisition unit 502 is used to acquire a text token of the text to be processed and a number of candidate image tokens of the image to be processed, wherein the text token contains global text features for indicating task intent; The calculation unit 504 is used to calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent. The filtering unit 506 is used to filter target image tokens from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree. The target image token is concatenated with the text token and input into the multimodal inference model so that the multimodal inference model can perform inference calculations on the target image token based on the task intent.

[0067] In some embodiments, calculating the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: Using the global text features as the query vector, each candidate image token in the plurality of candidate image tokens is used as the key vector and the value vector in turn. Attention weights are calculated using the cross attention mechanism to obtain the attention weight of the query vector on each candidate image token in the plurality of candidate image tokens. Multiplying the attention weight by the value vector of the candidate image token yields the semantic relevance between the candidate image token and the task intent.

[0068] In some embodiments, the plurality of candidate image tokens maintain the spatial structure of the image to be processed, and the calculation of the spatial matching degree between the spatial location of each candidate image token and the task intent includes: Pooling the candidate image tokens yields global image features that characterize the overall semantics of the image. The global text features and the global image features are fused to obtain multimodal global fusion features; The multimodal global fusion features are input into a pre-trained spatial matching degree prediction network so that the output of the spatial matching degree prediction network is consistent with the spatial structure dimension of the image to be processed. Each element value in the spatial probability mask represents the spatial matching degree between the candidate image token at the corresponding spatial location and the task intent.

[0069] In some embodiments, training the spatial matching degree prediction network includes: Based on the inference loss output by the multimodal inference model and the regularization loss of the spatial matching degree prediction network, a supervision signal for the spatial matching degree prediction network is constructed. Based on the gradient of the supervision signal, the parameters of the spatial matching degree prediction network are updated through the backpropagation algorithm until the spatial matching degree prediction network converges.

[0070] In some embodiments, filtering target image tokens from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree includes: The semantic correlation degree between the semantic features contained in each candidate image token and the semantic intent of the task is calculated, and the element-wise product operation is performed with the spatial probability mask of the spatial location corresponding to the image token to obtain the comprehensive correlation degree of the image token. The candidate image tokens are sorted according to the overall relevance of each candidate image token; Among the sorted candidate image tokens, the candidate image tokens with the highest overall relevance are selected as the target image tokens.

[0071] In some embodiments, the plurality of candidate image tokens contain deep image semantic representations, and before calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent, the apparatus further includes: The input unit 508 is used to concatenate the plurality of candidate image tokens and the text tokens and input them into the large language model to obtain the plurality of candidate image tokens containing shallow image semantic representations and the text tokens containing shallow text semantic representations output by the large language model. The extraction unit 510 is used to extract deep semantics by using a text token containing shallow text semantic representation as a query vector and each candidate image token containing shallow image semantic representation as a key vector and a value vector, respectively, and using a cross-attention mechanism to extract deep semantics to generate several candidate image tokens containing deep image semantic representation.

[0072] In some embodiments, calculating the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: The generated candidate image tokens containing deep image semantic representations are input into a multilayer perceptron mapping network, so that the multilayer perceptron mapping network maps the deep image semantic representations to the prediction distribution space, and obtains the prediction score corresponding to each candidate image token among the candidate image tokens containing deep image semantic representations. Among them, the prediction score corresponding to each candidate image token in the several candidate image tokens containing deep image semantic representation represents the semantic relevance between the semantic features contained in the candidate image token and the task intent.

[0073] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0074] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0075] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or entity, or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0076] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0080] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0081] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0082] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0083] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0084] The above description is merely a preferred embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of one or more embodiments of this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this disclosure should be included within the scope of protection of one or more embodiments of this disclosure.

Claims

1. An image token filtering method, characterized in that, The method includes: Obtain a text token for the text to be processed and several candidate image tokens for the image to be processed, wherein the text token contains global text features for indicating the task intent; Calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent. Based on the semantic relevance and spatial matching degree, the target image token is filtered from the candidate image tokens; The target image token is concatenated with the text token and input into the multimodal inference model so that the multimodal inference model can perform inference calculations on the target image token based on the task intent.

2. The method according to claim 1, characterized in that, The calculation of the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: Using the global text features as the query vector, each candidate image token in the plurality of candidate image tokens is used as the key vector and the value vector in turn. Attention weights are calculated using the cross attention mechanism to obtain the attention weight of the query vector on each candidate image token in the plurality of candidate image tokens. Multiplying the attention weight by the value vector of the candidate image token yields the semantic relevance between the candidate image token and the task intent.

3. The method according to claim 1, characterized in that, The plurality of candidate image tokens maintain the spatial structure of the image to be processed, and the calculation of the spatial matching degree between the spatial location of each candidate image token and the task intent includes: Pooling is performed on the candidate image tokens to obtain global image features that characterize the overall semantics of the image; The global text features and the global image features are fused to obtain multimodal global fusion features; The multimodal global fusion features are input into a pre-trained spatial matching degree prediction network so that the output of the spatial matching degree prediction network is consistent with the spatial structure dimension of the image to be processed. Each element value in the spatial probability mask represents the spatial matching degree between the candidate image token at the corresponding spatial location and the task intent.

4. The method according to claim 3, characterized in that, Training the spatial matching degree prediction network includes: Based on the inference loss output by the multimodal inference model and the regularization loss of the spatial matching degree prediction network, a supervision signal for the spatial matching degree prediction network is constructed. Based on the gradient of the supervision signal, the parameters of the spatial matching degree prediction network are updated through the backpropagation algorithm until the spatial matching degree prediction network converges.

5. The method according to claim 3, characterized in that, The step of filtering target image tokens from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree includes: The semantic correlation degree between the semantic features contained in each candidate image token and the semantic intent of the task is calculated, and the element-wise product operation is performed with the spatial probability mask of the spatial location corresponding to the image token to obtain the comprehensive correlation degree of the image token. The candidate image tokens are sorted according to the overall relevance of each candidate image token; Among the sorted candidate image tokens, the candidate image tokens with the highest overall relevance are selected as the target image tokens.

6. The method according to claim 1, characterized in that, The candidate image tokens contain deep image semantic representations. Before calculating the semantic relevance between the semantic features contained in each candidate image token and the task intent, the method further includes: The candidate image tokens and the text tokens are concatenated and input into the large language model to obtain the candidate image tokens and the text tokens with shallow image semantic representations output by the large language model. Using text tokens containing shallow text semantic representations as query vectors, and each candidate image token containing shallow image semantic representations as key vector and value vector respectively, deep semantic extraction is performed using cross-attention mechanism to generate several candidate image tokens containing deep image semantic representations.

7. The method according to claim 6, characterized in that, The calculation of the semantic relevance between the semantic features contained in each candidate image token and the semantic intent of the task includes: The generated candidate image tokens containing deep image semantic representations are input into a multilayer perceptron mapping network, so that the multilayer perceptron mapping network maps the deep image semantic representations to the prediction distribution space, and obtains the prediction score corresponding to each candidate image token among the candidate image tokens containing deep image semantic representations. Among them, the prediction score corresponding to each candidate image token in the several candidate image tokens containing deep image semantic representation represents the semantic relevance between the semantic features contained in the candidate image token and the task intent.

8. An image token screening device, characterized in that, The device includes: The acquisition unit is used to acquire a text token of the text to be processed and a number of candidate image tokens of the image to be processed, wherein the text token contains global text features for indicating the task intent; The computing unit is used to calculate the semantic correlation between the semantic features contained in each candidate image token and the task intent, and the spatial matching degree between the spatial location of each candidate image token and the task intent. A filtering unit is used to filter target image tokens from the plurality of candidate image tokens based on the semantic relevance and the spatial matching degree. The target image token is concatenated with the text token and input into the multimodal inference model so that the multimodal inference model can perform inference calculations on the target image token based on the task intent.

9. An electronic device, characterized in that, It includes a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by invoking the machine-readable instructions.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-readable instructions, which, when invoked and executed by a processor, implement the method described in any one of claims 1 to 7.

Images