Multimodal feature interaction method and apparatus based on information filtering
By introducing semantically guided adaptive information filtering and bidirectional interaction mechanisms into multimodal analysis, the problem of spurious associations caused by noise interference is solved, and the discriminative performance and robustness of the model are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2026-05-08
- Publication Date
- 2026-07-24
AI Technical Summary
Existing multimodal analysis methods tend to incorporate noise along with valid information into the model when dealing with complex scenarios, leading to the model learning spurious associations that are irrelevant to the task, resulting in insufficient robustness and generalization ability.
By introducing semantically guided adaptive information filtering, adaptive filtering weights are generated to filter noise in image features. Furthermore, cross-modal information is verified and enhanced through a two-way closed-loop interaction mechanism between image features and text features.
It significantly improves the model's discriminative performance and robustness, and enhances its generalization ability under conditions of limited samples or changing scenarios.
Smart Images
Figure CN122173771B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimodal data processing technology, and in particular to a multimodal feature interaction method and apparatus based on information filtering. Background Technology
[0002] In many applications of artificial intelligence, such as weather forecasting, intelligent security, and medical image analysis, systems need to simultaneously understand and make decisions based on data from different modalities. Multimodal intelligent analysis technology aims to fuse this heterogeneous information to obtain a more comprehensive and accurate understanding than a single modality.
[0003] Currently, common multimodal analysis methods typically follow a process of "feature extraction - feature fusion - task prediction". In the feature extraction stage, separate encoders are used to map data from different modalities into vector representations. In the feature fusion stage, concatenation, weighted summation, or attention-based interactive methods are typically used to combine features from different modalities to form a unified joint representation.
[0004] However, existing technologies face significant challenges when dealing with complex real-world scenarios. Taking weather forecast data analysis as an example, the input image data often contains a large amount of irrelevant interference information such as environmental clutter and equipment noise; while the corresponding text data may contain vague wording, semantic redundancy, or incomplete descriptions. Traditional feature fusion methods incorporate noise along with valid information into the model, leading to the model learning spurious associations irrelevant to the task. Furthermore, the interaction method based on unidirectional attention lacks a reverse verification mechanism, making it prone to cumulative bias due to errors in a particular modality. This results in insufficient robustness and generalization ability of the model when faced with limited training data, varying scene distributions, or noise interference. Summary of the Invention
[0005] Therefore, it is necessary to provide a multimodal feature interaction method and apparatus based on information filtering to address the aforementioned technical problems.
[0006] A multimodal feature interaction method based on information filtering, the method comprising: Acquire image and text data, and extract multiple local image features and text feature representations respectively; Construct semantic conditions based on the text feature representation; Guided by the aforementioned semantic conditions, corresponding adaptive filtering weights are generated for each local feature of the image; The local features of the image are weighted and filtered using the adaptive filtering weights to obtain the filtered image features. Using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain the corrected text features; Using the corrected text features as secondary guiding conditions, the filtered image features are updated a second time to obtain the final image features.
[0007] In one embodiment, the method further includes: defining the relevance score of the text guidance as: ; in, For trainable projection matrices, d As a dimension, Indicates the first j Local features of an image With current semantic conditions The correlation strength, The number of local features in the image; For each local feature of the image, the corresponding adaptive filtering weights are generated as follows: ; in, For activation function, For temperature coefficient, This is a bias term.
[0008] In one embodiment, the method further includes: using text features as a query vector, using the filtered set of image features as a key vector and a value vector, and updating the text features using the following relevance mapping formula: ; ; ; Where A is the cross-modal correlation matrix. For text query matrix, The image feature key matrix, The number of text features. For the number of semantic conditions, The image feature value matrix, Represents the original text feature matrix. This represents the updated text feature matrix. This is the corrected text feature matrix.
[0009] In one embodiment, the method further includes: performing a second update on the filtered image features using the corrected text feature matrix as follows: ; ; ; ; in, This represents the relevance score based on the corrected text feature matrix. , This represents a trainable projection matrix. Indicates the first j Local features of the filtered image and , For secondary guiding weight, for Visual features obtained after secondary guidance It is by indivual The combined visual feature matrix and They are respectively Temperature coefficient and bias term, Represents the final image features. This represents the number of local features in the image.
[0010] In one embodiment, during model training, the generation process of the adaptive filtering weights, the first update process, and the second update process are jointly optimized using a loss function that includes a filtering regularization term.
[0011] In one embodiment, the filtering regularization term in the loss function includes an average retention rate constraint term; the average retention rate constraint term is: ; in, Represents the similarity function. Indicates the temperature coefficient. For the first i Image feature vectors after filtering of training samples Is with The corresponding matched text feature vector, Is with Irrelevant text feature vectors N The total number of samples.
[0012] In one embodiment, the loss function further includes: sparsity constraints and scaling constraints; The proportional constraint is as follows: ; The sparse constraint is: ; in, For adaptive filtering weights, The target average retention rate.
[0013] A multimodal feature interaction device based on information filtering, the device comprising: The feature extraction module is used to acquire image data and text data, and extract multiple local image features and text feature representations respectively; The condition construction module is used to construct semantic conditions based on the text feature representation; The information filtering module is used to generate corresponding adaptive filtering weights for each local feature of the image, guided by the semantic conditions. The bidirectional interaction module is used to perform weighted filtering on local image features using the adaptive filtering weights to obtain filtered image features; using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain corrected text features; using the corrected text features as secondary guiding conditions, the filtered image features are updated for the second time to obtain the final image features.
[0014] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Acquire image and text data, and extract multiple local image features and text feature representations respectively; Construct semantic conditions based on the text feature representation; Guided by the aforementioned semantic conditions, corresponding adaptive filtering weights are generated for each local feature of the image; The local features of the image are weighted and filtered using the adaptive filtering weights to obtain the filtered image features. Using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain the corrected text features; Using the corrected text features as secondary guiding conditions, the filtered image features are updated a second time to obtain the final image features.
[0015] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire image and text data, and extract multiple local image features and text feature representations respectively; Construct semantic conditions based on the text feature representation; Guided by the aforementioned semantic conditions, corresponding adaptive filtering weights are generated for each local feature of the image; The local features of the image are weighted and filtered using the adaptive filtering weights to obtain the filtered image features. Using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain the corrected text features; Using the corrected text features as secondary guiding conditions, the filtered image features are updated a second time to obtain the final image features.
[0016] The aforementioned information filtering-based multimodal feature interaction method and apparatus suppresses noise in image features at the source by introducing semantically guided adaptive information filtering before multimodal interaction. Subsequently, a bidirectional closed-loop interaction mechanism is constructed between image features and text features, and between text features and image features, enabling mutual verification and enhancement of cross-modal information. This method allows the model to focus more on task-related essential clues when processing noisy, key-information-sparse multimodal data, thereby significantly improving the model's discriminative performance, robustness, and generalization ability under limited sample size or changing scenarios. Attached Figure Description
[0017] Figure 1 This is an application scenario diagram of a multimodal feature interaction method based on information filtering in one embodiment; Figure 2 This is a flowchart illustrating a multimodal feature interaction method based on information filtering in one embodiment; Figure 3 This is a structural block diagram of a multimodal feature interaction device based on information filtering in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0019] The multimodal feature interaction method based on information filtering provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers. Terminal 102 can collect or receive image data and text data from sensors and databases, and send them to server 104. Server 104 executes the method provided in this application to perform joint analysis and processing of multimodal data, and returns the analysis results to terminal 102. It is understood that this method can also be executed entirely locally on terminal 102 or other edge devices with computing capabilities.
[0020] In one embodiment, such as Figure 2As shown, a multimodal feature interaction method based on information filtering is provided, which is then applied to... Figure 1 Taking the server in the example, the following steps are included: Step 202: Obtain image data and text data, and extract multiple local image features and text feature representations respectively.
[0021] Step 204: Construct semantic conditions based on text feature representations.
[0022] To enhance the semantic guidance capability of the text and make it more suitable for the current task, this step constructs a stronger semantic condition. Specifically, a set of trainable semantic guidance vectors is introduced, which can be regarded as learnable cues. The semantic guidance vectors are concatenated before the text sequence to form a new input sequence, which is then processed through a lightweight fusion layer or directly through subsequent layers of the text encoder. Pooling is performed on the fused feature representation to generate a global semantic condition vector. This vector integrates task-specific prior knowledge and the text semantics of the current instance and will serve as a guiding signal for subsequent information filtering.
[0023] Step 206: Guided by semantic conditions, generate corresponding adaptive filtering weights for each local feature of the image.
[0024] This step, one of the core innovations, aims to filter out noise and irrelevant parts of image features using textual semantics. First, the relevance of each local image feature to the semantic condition is calculated. Visual features and semantic conditions are mapped to the same comparable space using two trainable projection matrices, and then the dot product is calculated as the relevance score, enabling the model to learn which visual patterns are relevant to the current semantics. A higher score indicates a greater relevance between the image region and the textual description.
[0025] Then, the score is converted into a filtering weight between 0 and 1. Finally, the calculated filtering weight is used to weight the local features of the original image element by element to obtain the filtered image features: features in regions with weights close to 0 are significantly suppressed, while features in regions with weights close to 1 are preserved and highlighted. In this way, a large amount of background noise, clutter, and other information that is irrelevant to the current semantics is effectively weakened before entering deep interaction.
[0026] Step 208: Use adaptive filtering weights to perform weighted filtering on local image features to obtain filtered image features.
[0027] After filtering, cleaner and more focused visual evidence is obtained. This step uses visual evidence to correct potentially incomplete or ambiguous semantic representations of the text.
[0028] Step 210: Using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain the corrected text features.
[0029] After the first update, the text semantics have incorporated reliable visual evidence, theoretically making them more accurate than the initial visual evidence. This step utilizes this more accurate semantics to further refine the filtered image features.
[0030] Step 212: Using the corrected text features as a secondary guiding condition, the filtered image features are updated a second time to obtain the final image features.
[0031] In the aforementioned information filtering-based multimodal feature interaction method, semantically guided adaptive information filtering is introduced before multimodal interaction to suppress noise in image features at the source. Subsequently, a bidirectional closed-loop interaction mechanism is constructed between image features and text features, and between text features and image features, to achieve mutual verification and enhancement of cross-modal information. This method enables the model to focus more on task-related essential clues when processing noisy multimodal data with sparse key information, thereby significantly improving the model's discriminative performance, robustness, and generalization ability under limited samples or changing scenarios.
[0032] In one embodiment, the relevance score of the text guidance is defined as: ; in, For trainable projection matrices, d As a dimension, Indicates the first j Local features of an image With current semantic conditions The higher the correlation strength, the more relevant the image region is to the current text semantics; This represents the number of local features in the image.
[0033] Relevance score Mapped to a filter weight between 0 and 1 To achieve differentiable adaptive selection, adaptive filtering weights are generated for each local image feature as follows: ; in, For activation function, For temperature coefficient, This is a bias term.
[0034] In this way, the model can dynamically and adaptively assign appropriate filtering intensity to each image region based on the specific content of each pair of image and text data, thereby preserving and enhancing key visual information that is semantically relevant while suppressing irrelevant noise.
[0035] In one embodiment, text features are used as the query vector, and the filtered set of image features are used as the key vector and value vector. The text features are updated using the following relevance mapping formula: ; ; ; Where A is the cross-modal correlation matrix. For text query matrix, The image feature key matrix, The number of text features. For the number of semantic conditions, The image feature value matrix, Represents the original text feature matrix. This represents the updated text feature matrix. This is the corrected text feature matrix.
[0036] In this embodiment, the purpose of the first update is to use the reliable visual evidence provided by the filtered image features to correct and enhance the semantics of text that may be incomplete or ambiguous. This process enables the text features to obtain information from the most relevant, filtered visual regions, thereby correcting and completing the semantics of the text and improving the consistency of cross-modal representation.
[0037] In one embodiment, the modified text feature matrix is used to perform a second update on the filtered image features as follows: ; ; ; ; in, This represents the relevance score based on the corrected text feature matrix. , This represents a trainable projection matrix. Indicates the first j Local features of the filtered image and , For secondary guiding weight, for Visual features obtained after secondary guidance It is by indivual The combined visual feature matrix and They are respectively Temperature coefficient and bias term, Represents the final image features. This represents the number of local features in the image.
[0038] In this embodiment, the second update is a crucial step in forming a two-way interactive closed loop. Its purpose is to utilize the more accurate textual semantics obtained after the first update to refocus and refine the filtered image features. Through this step, the model achieves a feedback loop: visual features help correct the text, and the corrected text, in turn, guides the further purification and highlighting of visual features with more accurate semantics. This effectively reduces the error accumulation that may result from one-way interaction and makes the model more sensitive and consistent in its perception of key cues.
[0039] In one embodiment, during model training, the generation process of adaptive filtering weights, the first update process, and the second update process are jointly optimized by using a loss function that includes a filtering regularization term.
[0040] In this embodiment, to ensure the information filtering module and the bidirectional interaction module work together and to prevent the filtering module from failing, a specialized joint optimization strategy is required during training. Total loss function Designed as a weighted sum of multiple losses: ; It is a task-specific loss that ensures the model output conforms to the supervision signal.
[0041] It is an optional alignment loss used to bring matching text and image features closer together and push away mismatched features, thereby enhancing cross-modal consistency.
[0042] It is the key filtering regularization term, which directly affects the adaptive filtering weights. It is the core of stable filtering behavior and enabling collaboration with interaction modules.
[0043] Through backpropagation, the loss function simultaneously updates the parameters of the information filtering module, the bidirectional interaction module, and the task prediction module. This end-to-end joint optimization enables the filtering module to learn to provide high-quality input for subsequent interactions, while the good performance of the interaction module provides learning signals for the filtering module through task and alignment losses. The two mutually promote each other, forming a high-performance organic whole.
[0044] In one embodiment, the average retention rate constraint is: ; in, Represents the similarity function. Indicates the temperature coefficient. For the first i Image feature vectors after filtering of training samples Is with The corresponding matched text feature vector, Is with Irrelevant text feature vectors N The total number of samples.
[0045] In one embodiment, the scaling constraint is: ; The sparsity constraint is: ; in, For adaptive filtering weights, The target average retention rate.
[0046] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0047] In one embodiment, such as Figure 3 As shown, a multimodal feature interaction device based on information filtering is provided, including: a feature extraction module 302, a condition construction module 304, an information filtering module 306, and a two-way interaction module 308, wherein: The feature extraction module 302 is used to acquire image data and text data, and extract multiple local image features and text feature representations respectively; Condition construction module 304 is used to construct semantic conditions based on the text feature representation; Information filtering module 306 is used to generate corresponding adaptive filtering weights for each local feature of the image, guided by the semantic conditions. The bidirectional interaction module 308 is used to perform weighted filtering on local image features using the adaptive filtering weights to obtain filtered image features; use the filtered image features as visual evidence to perform a first update on the text feature representation to obtain corrected text features; and use the corrected text features as a secondary guiding condition to perform a second update on the filtered image features to obtain the final image features.
[0048] In one embodiment, the information filtering module 306 is further configured to define the relevance score of the text guidance as follows: ; in, For trainable projection matrices, d As a dimension, Indicates the first j Local features of an image With current semantic conditions The correlation strength, The number of local features in the image; For each local feature of the image, the corresponding adaptive filtering weights are generated as follows: ; in, For activation function, For temperature coefficient, This is a bias term.
[0049] In one embodiment, the bidirectional interaction module 308 is further configured to use text features as query vectors and the filtered set of image features as key and value vectors, and update the text features using the following relevance mapping formula: ; ; ; Where A is the cross-modal correlation matrix. For text query matrix, The image feature key matrix, The number of text features. For the number of semantic conditions, The image feature value matrix, Represents the original text feature matrix. This represents the updated text feature matrix. This is the corrected text feature matrix.
[0050] In one embodiment, the two-way interaction module 308 is further configured to perform a second update on the filtered image features using the corrected text feature matrix: ; ; ; ; in, This represents the relevance score based on the corrected text feature matrix. , This represents a trainable projection matrix. Indicates the first j Local features of the filtered image and , For secondary guiding weight, for Visual features obtained after secondary guidance It is by indivual The combined visual feature matrix and They are respectively Temperature coefficient and bias term, Represents the final image features. This represents the number of local features in the image.
[0051] In one embodiment, during model training, the generation process of the adaptive filtering weights, the first update process, and the second update process are jointly optimized using a loss function that includes a filtering regularization term.
[0052] In one embodiment, the filtering regularization term in the loss function includes an average retention rate constraint term; the average retention rate constraint term is: ; in, Represents the similarity function. Indicates the temperature coefficient. For the first i Image feature vectors after filtering of training samples Is with The corresponding matched text feature vector, Is with Irrelevant text feature vectors N The total number of samples.
[0053] In one embodiment, the loss function further includes: sparsity constraints and scaling constraints; wherein the scaling constraints are: ; The sparse constraint is: ; in, For adaptive filtering weights, The target average retention rate.
[0054] Specific limitations regarding the information filtering-based multimodal feature interaction device can be found in the limitations of the information filtering-based multimodal feature interaction method described above, and will not be repeated here. Each module in the aforementioned information filtering-based multimodal feature interaction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0055] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores training data, model parameters, etc. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the above method embodiments.
[0056] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0057] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.
[0058] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0060] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0061] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A multimodal feature interaction method based on information filtering, characterized in that, The method includes: Acquire image and text data, and extract multiple local image features and text feature representations respectively; Construct semantic conditions based on the text feature representation; Guided by the aforementioned semantic conditions, corresponding adaptive filtering weights are generated for each local feature of the image; The local features of the image are weighted and filtered using the adaptive filtering weights to obtain the filtered image features. Using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain the corrected text features; Using the corrected text features as secondary guiding conditions, the filtered image features are updated a second time to obtain the final image features; Generate corresponding adaptive filtering weights for each local feature of the image, including: The relevance score of the defined text guidance is: in, For trainable projection matrices, d As a dimension, Indicates the first j Local features of an image With current semantic conditions The correlation strength, The number of local features in the image; For each local feature of the image, the corresponding adaptive filtering weights are generated as follows: in, For activation function, For temperature coefficient, For bias terms; The first update to the text feature representation includes: Using text features as the query vector and the filtered image feature set as the key and value vectors, the text features are updated using the following relevance mapping formula: Where A is the cross-modal correlation matrix. For text query matrix, The image feature key matrix, The number of text features. For the number of semantic conditions, The image feature value matrix, Represents the original text feature matrix. This represents the updated text feature matrix. This is the corrected text feature matrix; The second update of the filtered image features also includes: The modified text feature matrix is then used to perform a second update on the filtered image features: in, This represents the relevance score based on the corrected text feature matrix. , This represents a trainable projection matrix. Indicates the first j Local features of the filtered image and , For secondary guiding weight, for Visual features obtained after secondary guidance It is by indivual The combined visual feature matrix and They are respectively Temperature coefficient and bias term, Represents the final image features. This represents the number of local features in the image.
2. The method according to claim 1, characterized in that, During model training, the generation process of adaptive filtering weights, the first update process, and the second update process are jointly optimized by using a loss function that includes a filtering regularization term.
3. The method according to claim 2, characterized in that, The filtering regularization term in the loss function includes an average retention rate constraint term; The average retention rate constraint is: in, Represents the similarity function. Indicates the temperature coefficient. For the first i Image feature vectors after filtering of training samples Is with The corresponding matched text feature vector, Is with Irrelevant text feature vectors N The total number of samples.
4. The method according to claim 3, characterized in that, The loss function also includes: sparsity constraints and scaling constraints; The proportional constraint is as follows: The sparse constraint is: in, For adaptive filtering weights, The target average retention rate.
5. A multimodal feature interaction device based on information filtering, characterized in that, The apparatus for implementing the information filtering-based multimodal feature interaction method according to any one of claims 1 to 4, the apparatus comprising: The feature extraction module is used to acquire image data and text data, and extract multiple local image features and text feature representations respectively; The condition construction module is used to construct semantic conditions based on the text feature representation; The information filtering module is used to generate corresponding adaptive filtering weights for each local feature of the image, guided by the semantic conditions. The bidirectional interaction module is used to perform weighted filtering on local image features using the adaptive filtering weights to obtain filtered image features; using the filtered image features as visual evidence, the text feature representation is updated for the first time to obtain corrected text features; using the corrected text features as secondary guiding conditions, the filtered image features are updated for the second time to obtain the final image features.