Image-text retrieval method based on adaptive fine-grained alignment and text consensus constraint

Abstract

The application discloses a kind of based on self-adapting fine-grained alignment and text consensus constraint's graph-text retrieval method, belong to computer vision, remote sensing interpretation and artificial intelligence field.This method is based on double-flow visual language coding framework, jointly extract image global feature, local image block feature and text global feature, and establish stable graph-text matching relationship by global contrast learning.Aiming at the problem of unstable local region selection caused by complex background and scale change, the method adaptively determines the key region according to the similarity of noun-image block and information entropy, generates a mask combined with spatial aggregation, and constructs a fine-grained alignment loss to suppress background interference.Aiming at the semantic difference problem of one image with multiple texts, the text description group is constructed according to the image, the soft gate consistency constraint is introduced, the influence of high consensus text is enhanced and the influence of low consensus text is weakened, so as to improve the precision and robustness of text retrieval image and image retrieval text task in remote sensing scene.

Description

Technical Field

[0001] This invention relates to a text and image retrieval method based on adaptive fine-grained alignment and text consensus constraints, belonging to the fields of remote sensing interpretation, computer vision and artificial intelligence technology. Background Technology

[0002] With the continuous growth of remote sensing image data, using natural language descriptions for remote sensing image retrieval has become an important application direction for remote sensing interpretation and intelligent retrieval. Compared with natural images, remote sensing images typically have complex background structures, dense small targets, and similar appearances of similar scenes. However, text descriptions are often mainly scene-level summaries, making it difficult to maintain a stable correspondence between the local semantics of the image and the semantics of the text nouns. At the same time, the same image often corresponds to multiple descriptions, and different descriptions have different focuses on the scene, targets, and spatial relationships, which can easily lead to scattered textual representations of the same image.

[0003] Currently, most remote sensing image-text retrieval methods are based on a visual-linguistic dual-coding framework, achieving global alignment between images and text by constructing a shared embedding space and employing contrastive learning. To enhance local matching capabilities, existing methods typically introduce fine-grained alignment at the image patch or local region level, but often employ a "fixed selection" approach when filtering key regions. "Fixed selection" refers to pre-setting a uniform number or proportion of elements to retain before filtering, first calculating the similarity or response value between text semantic units and each image patch, then sorting them from high to low according to the similarity or response value, and selecting the top-ranked elements. k Image blocks or fixed proportions r Image patches are treated as key regions without being adjusted according to changes in the current text semantics and the distribution of image patch responses. This approach tends to introduce redundant background patches when responses are concentrated, and may miss true key regions when responses are dispersed, making it difficult to adapt to situations where there are significant differences in target scale, salience, and background complexity in remote sensing scenes.

[0004] Furthermore, a single remote sensing image often corresponds to multiple text descriptions, and these descriptions frequently differ in their semantic emphasis. Existing methods typically train multiple texts for the same image as independent samples, or perform simple averaging or summing of the features of multiple texts. The former fails to effectively utilize the inherent connections between texts within the same image, easily leading to fragmented text representations; the latter lacks the ability to distinguish differences in consistency within groups, making it difficult to balance text diversity and representation stability.

[0005] Therefore, existing technologies still lack a text-image retrieval method that can adaptively select key regions based on text semantics and image patch response distribution, and can also impose consistency constraints on multiple text descriptions corresponding to the same image, thereby improving the accuracy and stability of text-image retrieval in complex remote sensing scenarios. Summary of the Invention

[0006] To address the challenges of inaccurate fine-grained semantic targeting in complex backgrounds and the fragmented text representations and unstable training signals caused by multiple texts per image in cross-modal remote sensing image retrieval, this invention proposes an image-text retrieval method based on adaptive fine-grained alignment and text consensus constraints. This method, building upon the global alignment of a visual-linguistic dual-encoding framework, introduces two mechanisms: adaptive key image patch selection and consensus gating for multiple texts within the same image. On one hand, it adaptively filters semantically relevant regions under different scales and complex backgrounds, reducing local alignment noise. On the other hand, it models the intra-group consistency of multiple texts within the same image and dynamically adjusts the constraint strength, thereby improving the stability of text representation and the robustness of cross-modal matching. Ultimately, this achieves a performance improvement in bidirectional remote sensing image-text retrieval that requires no additional annotation and is easily integrated.

[0007] The technical solution of the present invention includes the following steps:

[0008] (1) Multimodal input processing: acquire remote sensing images and their corresponding text descriptions, input them into the image encoder and text encoder respectively, obtain global image features and global text features, and normalize the global image features and global text features to calculate image-text similarity.

[0009] (2) Global semantic alignment: In the shared embedding space, global image-text comparison learning is used as the main supervision signal, and triple constraints are introduced to widen the positive and negative sample intervals, thereby enhancing the discriminativeness and consistency of image-text matching at the global level.

[0010] (3) Fine-grained alignment processing: extract semantic units of nouns from the text description and encode them into noun features; extract image patch features from the image encoder; calculate the similarity between noun features and each image patch feature; normalize the similarity to obtain a weight distribution; calculate the information entropy based on the weight distribution and determine the retention ratio of candidate image patches accordingly; filter candidate image patches; refine the spatial connectivity of candidate image patches to obtain the final mask; divide the image patch into positive sample regions and negative sample regions based on the final mask; and construct a fine-grained alignment loss.

[0011] (4) Consensus constraint processing for multiple texts in the same image: In a batch, multiple text descriptions corresponding to the same image are grouped according to the image identifier, the consistency score of the text features in each description group is calculated, and the gating weight is determined according to the consistency score. A consensus center is constructed using the text features in the description group, and a consistency regularization that converges to the consensus center is applied to each text feature according to the gating weight.

[0012] (5) Joint training and retrieval output: The global alignment loss, fine-grained alignment loss and text consensus regularization are weighted and summed to form the total loss for training. In the inference stage, cross-modal similarity is calculated based on the shared embedding space after training to realize bidirectional retrieval of text to image and image to text.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0014] 1. Improve the reliability of fine-grained matching: Through information entropy-driven dynamic key block filtering and spatial focusing, it can adaptively locate semantically related regions, reduce local alignment noise caused by complex backgrounds and scale changes, and improve the fine-grained distinguishability between similar scenes.

[0015] 2. Enhance the robustness of "one image, multiple texts": By using intra-group consistency estimation and gating consensus regularization, it is possible to suppress text representation dispersion and semantic drift, and improve the stability of cross-modal training signals;

[0016] 3. Easy to integrate and highly adaptable: This method can be seamlessly embedded into existing dual-coding retrieval frameworks without relying on additional annotation information, and can stably improve bidirectional retrieval performance on typical remote sensing image and text retrieval data. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the principle of image and text retrieval based on adaptive fine-grained alignment and text consensus constraints in this invention.

[0018] Figure 2 : This is a schematic diagram of the adaptive fine-grained alignment module of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] like Figure 1As shown, this invention addresses cross-modal retrieval tasks of remote sensing images and text. It is an image-text semantic alignment method based on the CLIP dual-stream coding structure and a parameter fine-tuning strategy, targeting both "image retrieval by text" and "text retrieval by image" tasks. It enhances fine-grained discrimination capabilities while maintaining inference efficiency. The framework is based on an image encoder and a text encoder, fine-tuning only the lightweight Adapter layer to achieve remote sensing domain adaptation. Furthermore, it introduces two key modules based on global image-text comparison learning: an Adaptive Patch Selection (APS) module and a Consensus-Gated Caption Regularization (CGCR) module for multiple texts within the same image.

[0021] Given a remote sensing image and its corresponding text description This invention employs CLIP's two-stream coding framework to extract image and text feature vectors separately. Specifically, the image encoder... With text encoder Output global image and text feature vectors , Normalization is performed during the training phase to calculate cross-modal similarity. At the global level, contrastive learning loss is used. As the primary supervisor, triple constraints are introduced. This is to further widen the gap between positive and negative samples.

[0022] In addition to global alignment, this invention further enhances cross-modal matching from two levels: local semantic alignment and consistency of multiple texts within the same image. First, in the fine-grained alignment stage, local image representations are obtained from the image encoder, and noun-level semantic units are extracted from the text. By calculating the correlation distribution between nouns and image patches, image patches with stronger semantic association with nouns are adaptively selected for supervision, thereby highlighting effective local information and suppressing redundant noise in complex backgrounds. Based on the selected local patches and noun representations, a fine-grained alignment loss is constructed. This is used to constrain the consistency between semantic nouns and local image regions. Furthermore, CGCR estimates intra-group consensus for multiple descriptions of the same scene and generates gating weights, strengthening consistency constraints on high-consensus descriptions and moderately relaxing them on low-consensus descriptions, thus forming a text consensus regularization term. This improves the stability and robustness of text representation. The overall training objective is as follows:

[0023] (1)

[0024] in , , These are the weighting coefficients for each sub-loss, used to balance the contributions of each loss.

[0025] To alleviate the problem of "background dominance and lack of prominence of key areas" in remote sensing image retrieval, this invention designs an adaptive fine-grained alignment module (APS), such as... Figure 2 As shown, this module uses noun features extracted from the text as queries, adaptively selects the local regions most relevant to the semantics of the nouns within the image patch space, and constructs a fine-grained alignment loss accordingly, thereby strengthening the correspondence between "nouns and key regions" and suppressing background interference.

[0026] 1. The detailed steps for the adaptive fine-grained alignment module are as follows:

[0027] (1) Similarity calculation and weight normalization

[0028] Given noun features With image patch feature set First, calculate the similarity between the nouns and each image patch. response:

[0029] (2)

[0030] The similarity scores are then normalized using Softmax to obtain distributed weights:

[0031] (3)

[0032] in, Temperature coefficient. Weighting. Noun For image patches The relative level of attention.

[0033] (2) Information entropy measurement and dynamic retention ratio

[0034] After obtaining the relevance weight distribution of different terms to the image patch set, the core of APS lies in adaptively determining the size of the candidate positive region. Due to significant differences in target scale, salience, and background complexity in remote sensing images, the response distribution corresponding to different terms often exhibits different forms such as "sharp concentration" or "gentle dispersion." If a fixed top-k strategy is adopted, redundant background patches are easily introduced when the response is concentrated, and positive samples are missing when the response is dispersed, thereby weakening the reliability of fine-grained supervision. Therefore, this invention introduces information entropy to quantify the uncertainty of the weight distribution and dynamically adjusts the retention ratio accordingly. Specifically, the normalized information entropy of the weight distribution is first calculated:

[0035] (4)

[0036] in, For numerical stability terms, This represents the number of image patches. (Through...) After normalization, It can intuitively reflect the uncertainty of the distribution: when the weights are highly concentrated in a few image patches, A value close to 0 indicates a clear noun reference; when the weights are relatively evenly distributed across most image patches... A value close to 1 indicates a more dispersed response and greater location uncertainty.

[0037] Based on entropy The proportion of candidates retained Mapping to interval And further, the dynamic number of top-k values ​​is obtained:

[0038] (5)

[0039] in, and The minimum and maximum coverage areas are controlled separately. This design allows APS to adaptively adjust the selection scale according to the distribution pattern: when the noun response is more concentrated (low entropy), the retention ratio increases. The value is closer This results in sparser and more focused candidate regions; when the response is more dispersed (high entropy), the retention ratio... The value is closer This expands the scope of retention to avoid missing key areas.

[0040] (3) Top-k preliminary screening and spatial focusing optimization

[0041] The dynamic candidate size is obtained from information entropy. Then, APS first determines the weight distribution. Perform a top-k preliminary screening and select the one with the largest weight. Image patches form the initial candidate mask. This step can quickly eliminate most low-relevance regions, allowing subsequent fine-grained supervision to focus on local areas that are more likely to correspond to the semantics of the nouns. However, relying solely on weight size may still introduce spatially scattered high-response background blocks (such as road textures, building edges, or repetitive structures). These fragmented responses can lead to discontinuities in positive regions, thereby weakening the stability of local alignment.

[0042] To further enhance the spatial consistency and interpretability of the region, this invention introduces spatial focusing optimization based on the initial screening: Restored to The image patch mesh is processed, and maximum connected component refinement is performed, retaining only the largest connected regions to obtain the final mask. :

[0043] (6)

[0044] in, This represents the maximum connected component retention operation, which is used to remove spatially discrete small connected components from the initial candidate mask, retaining only the connected regions with the largest area, thereby reducing the interference of scattered high-response background blocks on subsequent fine-grained alignment.

[0045] (4) Loss of positive and negative region division and fine-grained alignment

[0046] Final binary mask Obtain the positive sample region Set and negative sample region Based on this set, average pooling is performed on the image patch features within the positive and negative regions respectively to obtain region-level representations. and Subsequently, a fine-grained alignment loss is constructed. Encourage noun features It is defined as being closer to positive region features and more separated from negative region features:

[0047] (7)

[0048] (8)

[0049] in, Represents cosine similarity. The temperature coefficient in fine-grained alignment. This represents the number of valid nouns extracted from the current image. This is determined by analyzing the set of positive sample regions. and negative sample region set The image patch features within the region are averaged using pooling to obtain a region-level representation. and And based on the region-level representation, a fine-grained alignment loss is constructed. The model can jointly optimize the correspondence between local semantics and image regions on all valid nouns, thereby enhancing the matching of key local regions and suppressing erroneous correspondences with negative regions.

[0050] In remote sensing cross-modal retrieval, the same image often corresponds to multiple text descriptions, and the semantic points emphasized by different descriptions are inconsistent (e.g., some descriptions emphasize local targets, while others emphasize global category attributes). This leads to significant dispersion in the text embeddings of the same image in the feature space, thus increasing the difficulty of image-text alignment. To address this, this invention proposes the Same Image Multi-Text Consensus Constraint Module (CGCR), which utilizes intra-group consistency estimation, gating weight allocation, and consensus center regularization to enhance the consistency and robustness of the same image text representation while preserving reasonable differences in description.

[0051] 2. The detailed steps for the multi-text consensus constraint module within the same graph are as follows:

[0052] (1) Construction of text collection within the group

[0053] Since the same image corresponds to multiple descriptions, aligning them line by line can easily overlook intra-group relationships and cause optimization conflicts. Therefore, within a batch, text groups for the same image are constructed based on the image sequence ID. Let there be a total of [number missing] descriptions within a batch. The image and text sample, for the first The sample image has the following ID: The corresponding global text features are (Output and normalized by the text encoder). Divide the text within the same image into several groups based on the image ID:

[0054] (9)

[0055] in, Representing an image The number of text entries appearing in the current batch corresponds to this grouping process. Figure 1 Input: Multiple text features of the same image.

[0056] (2) Within-group consistency estimation

[0057] Considering that not all texts containing the same image possess the same semantic consistency, CGCR first calculates the average similarity of texts within the group to measure consensus strength, defining the similarity between two texts as:

[0058] (10)

[0059] in, This represents the global features of the text. Simultaneously, the consistency score is obtained by averaging the pairwise similarities within each group. :

[0060] (11)

[0061] when When, it indicates that the multiple descriptions of the image are semantically consistent; when This indicates a greater discrepancy in the description or the presence of noisy description. This process corresponds to... Figure 1 "Intra-group text similarity calculation" in the middle.

[0062] (3) Consensus gating weight generation

[0063] To avoid imposing excessive contraction on low-consistency groups, CGCR adjusts the consistency score. Mapped to gate weights This adaptively adjusts the strength of intra-group consistency regularization. A threshold is introduced. and ,definition:

[0064] (12)

[0065] The aim is to make high-consensus groups more deserving of "tightening" of constraints, while low-consensus groups can be moderately relaxed, in order to avoid imposing excessive constraints on semantic disagreements or noisy descriptions. This process corresponds to the box... Figure 1 "Soft gating weight calculation" in Chinese.

[0066] (4) Consensus Center Construction and Regularized Loss

[0067] To obtain shared semantic anchors for texts within the same image, CGCR constructs a consensus center using the mean of text features within the group and applies a "convergence towards the center" consistency regularization to the texts within the group, defining the consensus center vector:

[0068] (13)

[0069] Subsequently, a consensus regularization rule that "converges towards the consensus center" is applied to each text within the group, and then... Adjusting the intensity:

[0070] (14)

[0071] This loss encourages multiple texts within the same graph to be more compact around the consensus center, thereby reducing within-group variance and improving the stability of text representations; the gating weights ensure that the constraint strength adapts to the consensus level, achieving a balance between consistency and diversity. This process corresponds to the box... Figure 1 "Consensus Center Loss Calculation" in China.

[0072] To verify the effectiveness of the method of the present invention, several representative CLIP-based remote sensing image and text retrieval methods were selected for comparative experiments, specifically:

[0073] Method 1: Full-FT CLIP proposed by Radford et al., see "Radford.; Alec.; et al. Learning transferable visual models from natural language supervision. In "International conference on machine learning. PmLR, 2021."

[0074] Method 2: PE-RSITR proposed by Yuan et al., see reference "Yuan Y.; Zhan Y.; Xiong Z. Parameter-efficient transfer learning for remote sensing image–text retrieval. In IEEE Transactions on Geoscience and Remote Sensing, 2023, 61:1-14."

[0075] Method 3: TDUF proposed by Yang et al., see reference "Yang M.; Chen L.; Jing NA tripletdictionary-driven learning and uncertainty-aware fusion for remote sensing image-text retrieval. In Expert Systems with Applications, 2025, 290:128454."

[0076] Method 4: MSSA proposed by Liao et al., see reference "Liao Y.; Hu Z.; Jin F.; et al. MSSA: A Multi-Scale Semantic-Aware Method for Remote Sensing Image–Text Retrieval. In Remote Sensing, 2025, 17(19): 3341."

[0077] Method 5: HarMA proposed by Huang, see reference "Huang T. Efficient remote sensing with harmonized transfer learning and modality alignment. In arXiv preprint arXiv:2404.18253, 2024."

[0078] Method 6: FGVLA proposed by Li et al., see reference "Li S.; Ji H.; Liu F.; et al. Fine-Grained Visual-Language Alignment for Remote Sensing Image-Text Retrieval. In IEEE Transactions on Geoscience and Remote Sensing, 2025."

[0079] As shown in Tables 1-3, this invention achieves overall improvements on three publicly available remote sensing cross-modal retrieval datasets. Taking RSICD (Table 1) as an example, compared to the FGVLA method, the R@1, R@5, and R@10 values ​​for image retrieval in the text direction are improved by 0.83%, 1.83%, and 0.82%, respectively, while the R@1, R@5, and R@10 values ​​for text retrieval in the image direction are improved by 0.14%, 0.11%, and 0.18%, respectively, resulting in a 0.65% improvement in mR. Overall, this method outperforms the compared methods such as Full-FT CLIP, PE-RSITR, and HarMA. On the RSICD and UCM datasets, the mR values ​​are improved by 0.93% and 1.24%, respectively. However, R@10 showed a slight decrease of 0.22% in image retrieval text in RSITMD and a decrease of 0.19% in text retrieval image in UCM. The possible reason for this phenomenon is that the similarity of the last sample in the top-10 is close and the ranking is more easily disturbed, but it does not affect the overall performance gain.

[0080] Table 1 compares the test results of various methods on the RSICD dataset.

[0081]

[0082] Table 2 Comparison of various methods on the RSITMD dataset

[0083]

[0084] Table 3. Comparison of various methods on the UCM dataset

[0085]

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-image retrieval method based on adaptive fine-grained alignment and text consensus constraints, characterized in that, Includes the following steps: (1) Multimodal input processing: acquire remote sensing images and their corresponding text descriptions, input them into the image encoder and text encoder respectively, obtain global image features and global text features, and normalize the global image features and global text features to calculate image-text similarity. (2) Global semantic alignment: In the shared embedding space, global image-text comparison learning is used as a supervision signal, and triple constraints are introduced to widen the positive and negative sample intervals, thereby enhancing the discriminativeness and consistency of image-text matching from a global perspective. (3) Fine-grained alignment processing: Noun semantic units are extracted from the text description and encoded as noun features. Image patch features are extracted from the image encoder. The similarity between noun features and each image patch feature is calculated. The similarity is normalized to obtain a weight distribution. The information entropy is calculated based on the weight distribution, and the retention ratio of candidate image patches is determined accordingly. Candidate image patches are screened, and the spatial connectivity of the candidate image patches is refined to obtain the final mask. Based on the final mask, the image patch is divided into positive sample regions and negative sample regions, and a fine-grained alignment loss is constructed. The information entropy of the above weight distribution... The information entropy is used to characterize the dispersion of the response of noun features across image patches, and to determine the retention ratio and number of candidate image patches based on the information entropy. The calculation formula is as follows: (1) in, For numerical stability terms, Number of image patches; via After normalization, It can intuitively reflect the uncertainty of the distribution: when the weights are highly concentrated in a few image patches, A value close to 0 indicates a clear noun reference; when the weights are relatively evenly distributed across most image patches... A value close to 1 indicates a more dispersed response and greater location uncertainty; (4) Same-image multi-text consensus constraint processing: Within a batch, multiple text descriptions corresponding to the same image are grouped according to image identifiers. The consistency score of text features within each description group is calculated, and a gating weight is determined based on the consistency score. A consensus center is constructed using the text features within the description group. This consensus center is obtained from the mean of each text feature within the same description group. A consistency regularization that converges to the consensus center is applied to each text feature according to the gating weight. In the above same-image multi-text consensus constraint processing, the consistency within a description group is determined by the pairwise similarity of text features within the same description group. The statistical measures are obtained, and the pairwise similarity within each group is calculated first: (2) in, The text uses global features, and the consistency score is obtained by averaging the pairwise similarities within each group. : (3) in, This indicates the number of text entries appearing in the image within the current batch; further, the consistency score is calculated. Mapped to gate weights To adaptively adjust the strength of intra-group consistency regularization; a threshold is introduced. and ,definition: (4) Among them, when When this occurs, it indicates that the multiple descriptions corresponding to the image have strong intra-group semantic consistency, and the gating weights... ;when When this occurs, it indicates that there are significant semantic differences between the descriptions or that there are noisy descriptions; gating weights are used. ;when At that time, gating weight With consistency score The strength of the consensus regularization within a group is increased as the consensus increases, in order to adaptively adjust the strength of the consensus regularization within the group. The purpose of this is to make high consensus groups more worthy of being tightened, while low consensus groups are moderately relaxed, so as to avoid imposing too strong constraints on semantic disagreements or noisy descriptions. (5) Joint training and retrieval output: The global alignment loss, fine-grained alignment loss and text consensus regularization are weighted and summed to form the total loss for training. In the inference stage, cross-modal similarity is calculated based on the shared embedding space after training to realize bidirectional retrieval of text to image and image to text.

2. The method according to claim 1, characterized in that, The image encoder and text encoder adopt a visual-language dual-stream structure. The image encoder is used to output global image features and image patch features. The text encoder is used to output global text features and noun features. During training, a parameter-efficient fine-tuning method is employed, freezing the backbone encoder parameters and updating only the adaptation module parameters; the total loss of the joint training is... Represented as: (5) The total loss consists of the global alignment loss. Fine-grained alignment loss Triple loss Consistency regularization loss We obtain the weighted summation; where, , , These are the weighting coefficients for each sub-loss, used to balance the contributions of each loss.

3. The method according to claim 2, characterized in that, In the fine-grained alignment process, the similarity between noun features and image patch features is calculated using cosine similarity, and then normalized using a temperature coefficient using Softmax to obtain the weight distribution of nouns on each image patch. The calculation formula is as follows: (6) (7) in, Indicates the first The noun and the first Similarity between features of image patches The normalized weights represent the nouns. For image patches The relative level of attention, It is expressed as a temperature coefficient.

4. The method according to claim 3, characterized in that, Spatial connectivity refinement is performed on candidate image patches, retaining only the largest connected regions as the final binary mask to improve the spatial coherence of key image patches and suppress discrete background responses; specifically, spatial focusing optimization is introduced: Restore to The image patch mesh is processed, and maximum connected component refinement is performed, retaining only the largest connected regions to obtain the final mask. : (8) in, This represents the maximum connected component retention operation, which is used to remove spatially discrete small connected components from the initial candidate mask, retaining only the connected regions with the largest area, thereby reducing the interference of scattered high-response background blocks on subsequent fine-grained alignment.

5. The method according to claim 4, characterized in that, According to the final binary mask Obtain the positive sample region Set and negative sample region Based on this set, average pooling is performed on the image patch features within the positive and negative regions respectively to obtain region-level representations. and Subsequently, a fine-grained alignment loss was constructed. Encourage noun features It is defined as being closer to positive region features and more separated from negative region features: (9) (10) in, Represents cosine similarity. The temperature coefficient in fine-grained alignment. This represents the number of valid nouns extracted from the current image; this is achieved by analyzing the set of positive sample regions. and negative sample region set The image patch features within the region are averaged using pooling to obtain a region-level representation. and And based on the region-level representation, a fine-grained alignment loss is constructed. The model can jointly optimize the correspondence between local semantics and image regions on all valid nouns, thereby enhancing the matching of key local regions and suppressing erroneous correspondences with negative regions.

6. The method according to claim 1, characterized in that, The consensus center is normalized to obtain a normalized consensus center; based on the normalized consensus center, consistency regularization constraints are applied to each text feature within the group, and the gating weights are utilized. The strength of the consistency regularization constraint is adjusted using the following formula: (11) (12) in, Indicates that the consensus center The consensus center obtained after normalizing it according to its own length The consistency regularization loss is used to encourage multiple texts in the same graph to be more compact around the consensus center, thereby reducing intra-group variance and improving the stability of text representations. The gating weight ensures that the constraint strength adapts to the consensus level, achieving a balance between consistency and diversity.

Images