Cross-modal retrieval method and system based on image block selection and dual-granularity alignment

Abstract

This application relates to a cross-modal retrieval method and system based on image patches and dual-granularity alignment. The method includes: acquiring image data and text data; performing mapping and global pooling followed by coarse-grained global semantic alignment, calculating self-supervised semantic reconstruction loss, and selecting a set of key image patches while generating a set of redundant image patches; calculating a bidirectional alignment matching score; obtaining a fine-grained semantic alignment loss based on the bidirectional alignment matching score; constructing a retrieval model, iteratively optimizing the retrieval model until convergence is achieved, and outputting the final cross-modal retrieval result. This invention combines coarse-grained global contrastive learning with fine-grained interaction. Global alignment ensures overall semantic consistency, while fine-grained alignment captures rich local details. The two complement each other through joint optimization, effectively solving the problem of single-granularity alignment easily getting trapped in local optima or semantic bias, and significantly improving the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to the fields of multimedia information processing, computer vision, and natural language processing, and more specifically, to a cross-modal retrieval method and system based on image patch selection and dual-granularity alignment. Background Technology

[0002] With the explosive growth of multimedia data, efficient retrieval between different modalities (such as images and text) has become a research hotspot in the field of artificial intelligence. Cross-modal retrieval aims to map heterogeneous visual and textual data to a unified semantic space through metric learning, thereby achieving "text-to-image search" or "image-to-text search." This technology is widely used in intelligent search engines, recommendation systems, and human-computer interaction scenarios.

[0003] Current cross-modal retrieval methods are mainly divided into two categories: coarse-grained alignment and fine-grained alignment. Coarse-grained alignment methods typically encode the entire image and the entire sentence into global feature vectors, and perform matching by calculating the similarity of global features. Although this method is computationally efficient, it often ignores the local semantic relationships between image regions and text words, resulting in insufficient retrieval accuracy when dealing with complex scenes containing rich details.

[0004] To address these issues, researchers have proposed fine-grained alignment methods, primarily including object detection-based methods and patch interaction methods based on the Visual Transformer (ViT). Early fine-grained methods relied on pre-trained object detectors to extract salient region features, but this approach is computationally expensive and limited by predefined object categories. In recent years, ViT-based methods have achieved significant performance improvements by dividing images into fixed-size patch sequences and performing fine-grained interactions with text tokens.

[0005] However, existing ViT-based fine-grained cross-modal retrieval methods still face serious challenges in practical applications, mainly in the following two aspects:

[0006] First, static patch selection strategies lack adaptability. Existing methods typically reserve a fixed number or proportion of visual patches for feature interaction. This "one-size-fits-all" strategy cannot adapt to changes in image scene complexity: for semantically dense complex scenes, too few patches will lead to the loss of key details; while for semantically sparse simple scenes, too many patches will introduce a large amount of redundant background noise, thereby interfering with the model's semantic understanding.

[0007] Second, single-granularity alignment mechanisms suffer from semantic bias. Existing fine-grained methods often overemphasize local patch-word interactions while neglecting global semantic consistency. This overfitting to local details can lead to the model "not seeing the forest for the trees," meaning that although some local features may be matched, the overall semantic context may be deviated, thus limiting the accuracy and robustness of retrieval. Summary of the Invention

[0008] Therefore, it is necessary to provide a cross-modal retrieval method and system based on image patch selection and dual granularity alignment that offers better retrieval accuracy to address the aforementioned technical problems.

[0009] Cross-modal retrieval methods based on image patch selection and dual-granularity alignment include: S1: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input the image data into the pre-trained visual encoder to obtain the visual image patch feature sequence, and input the text data into the pre-trained text encoder to obtain the text word feature sequence. S2: Map the visual image patch feature sequence and the text word feature sequence and perform global pooling to obtain the global image feature and the global text feature. Then perform coarse-grained global semantic alignment to obtain the global alignment loss. S3: Calculate the self-supervised semantic reconstruction loss based on the visual image patch feature sequence, select the key image patch set, and generate the redundant image patch set. S4: Calculate the bidirectional alignment matching score based on the key image patch set and the redundant image patch set; obtain the fine-grained semantic alignment loss based on the bidirectional alignment matching score; S5.: Construct a retrieval model based on global alignment loss, self-supervised semantic reconstruction loss, and fine-grained semantic alignment loss. Iteratively optimize the retrieval model until the convergence condition is met, and output the final cross-modal retrieval results.

[0010] Further, in step S1, the image data is input into a pre-trained visual encoder to obtain a visual image patch feature sequence. Specifically, the Vision Transformer is used as a pre-trained visual encoder to segment the image data into... From a fixed-size image patch, visual image patch feature sequences are extracted. ,in As category morphemes, For feature dimensions.

[0011] Further, in step S1, the text data is input into a pre-trained text encoder to obtain a text word feature sequence. Specifically, BERT is used as the text encoder to convert the text data into a word sequence, and the text word feature sequence is extracted. ,in This represents the text length.

[0012] Further, in step S2, the mapping and global pooling of the visual image patch feature sequence and text word feature sequence to obtain the global image features and global text features are specifically as follows: Construct a learnable linear projection layer and Through learnable linear projection layers and Map visual and textual features to a dimension of The common embedding space is then used to perform global pooling to obtain global image features and global text features.

[0013] Further, in step S2, coarse-grained global semantic alignment is performed, and the global alignment loss is obtained as follows: Minimize global contrast loss The calculation formula is as follows:

[0014] in, The image-to-text contrast loss is calculated using the following formula:

[0015] In the formula, For batch size, Indicates the first Global features of an image With the Text global features Cosine similarity between them Temperature coefficient; The text-to-image contrast loss is calculated in the same way as... correspond.

[0016] Furthermore, in step S3, specifically: S3.1: Utilizing visual image patch feature sequences With global features of the image Calculate the intramodal self-attention score To capture the saliency within the visual image; simultaneously, utilizing the feature sequences of visual image patches. With text global features Calculate cross-modal attention scores This is to introduce text semantic supervision; the specific calculation formula is as follows:

[0017]

[0018] In the formula, Indicates the scaling factor. This represents the normalization function, specifically using the Softmax function to map scores to the interval [0, 1]. Its definition is... Finally, the weighted fusion calculation is performed on the first... The final comprehensive saliency score of each image patch :

[0019] in, To balance the hyperparameters of intramodal attention and crossmodal attention; S3.2: The variance of the feature sequences of all visual image patches is used to measure the dispersion of the image semantic distribution. The calculation formula is as follows:

[0020] Higher variance indicates a wider semantic distribution, requiring the retention of more image patches; S3.3 Calculate the attention concentration of the image. The degree of attention focus is measured using the entropy of the comprehensive saliency score distribution. The calculation formula is as follows:

[0021] in To The result of Softmax normalization; higher entropy indicates distracted attention and uncertainty, requiring the retention of more image patches; S3.4 Calculate the dynamic retention rate Based on the semantic discreteness and attention concentration Adaptive calculation of image dynamic retention rate :

[0022] in, and These are the set lower and upper limits for retention rates, respectively. To balance the coefficients of the two complexity metrics, and These represent the semantic dispersion and attention concentration after batch normalization, respectively. S3.5 Dynamically Select Key Image Patches: Based on the calculated dynamic retention rate Total number of image blocks Determine the number to retain. Based on the comprehensive significance score calculated in step S3.1 Sort all image patches and select the highest-scoring ones. The key image patch set consists of several image patches. The remaining image blocks form a redundant image block set. ; S3.6: To ensure that the adaptively preserved image patches maintain the semantic integrity of the original image, a self-supervised semantic reconstruction loss is constructed. .

[0023] Further, in step S3.6, specifically: Self-supervised semantic reconstruction loss ; Calculate the global average pooling representation of the feature sequences of all visual image patches in the original image. ; Calculate the set of key image patches Global average pooling representation of feature sequences of all visual image patches in the dataset. The calculation formula is:

[0024] Calculate the mean squared error (MSE) between the two as the semantic reconstruction loss:

[0025] This loss function is used to constrain the dynamic selection process and prevent the loss of key semantic information.

[0026] Further, in step S4, specifically: S4.1 Feature Aggregation and Compensation: Defining a Set of Key Image Patches Set the aggregation ratio Number of target aggregation features The feature sequences of key visual image patches are mapped using an aggregation network composed of multilayer perceptrons, and the aggregation weight matrix is ​​calculated using the Softmax function. The calculation formula is:

[0027] Secondly, based on the weight matrix Weighted aggregation of key image patches yields Discriminative features :

[0028] Simultaneously, a context-aware fusion strategy is adopted, based on the set of redundant image patches. Calculate attention weights based on feature magnitudes to generate compensated features. :

[0029] Ultimately constitutes an enhanced visual representation ; S4.2 Fine-grained image patch-word alignment: Computationally enhancing visual representations Text features Image patch-Word similarity matrix between ,in The matching score is calculated using a bidirectional maximum correspondence mechanism.

[0030] Final two-way aligned total score .

[0031] Fine-grained semantic alignment loss The loss function is calculated using a two-way triplet, and the specific formula is as follows:

[0032]

[0033]

[0034] In the formula, For boundary parameters, Positive sample pairs The two-way alignment matching score; Indicates the image within the current batch. The most similar hard-to-bear sample text, Match a score to it; Indicates the text within the current batch The most similar hard-to-bear sample image. Match a score to it.

[0035] Further, in step S5, specifically: The overall objective function of the retrieval model It integrates global alignment loss, semantic reconstruction loss, and fine-grained semantic alignment loss, with the specific formulas defined as follows:

[0036] in: The global alignment loss is calculated based on the global feature similarity between the image and the text. This is the self-supervised semantic reconstruction loss, used to constrain semantic consistency during the dynamic selection process; This is the fine-grained semantic alignment loss.

[0037] Furthermore, this invention provides a cross-modal retrieval system based on image patch selection and dual-granularity alignment, including: Data acquisition and feature extraction module: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input image data into the pre-trained visual encoder to obtain visual image patch feature sequence, and input text data into the pre-trained text encoder to obtain text word feature sequence; Coarse-grained alignment module: Mapping and global pooling are performed on the visual image block feature sequence and the text word feature sequence to obtain global image features and global text features. Then, coarse-grained global semantic alignment is performed to obtain global alignment loss. Self-supervised semantic reconstruction loss module: Calculates self-supervised semantic reconstruction loss based on the feature sequence of visual image patches, selects the key image patch set, and generates a redundant image patch set. Fine-grained semantic alignment module: Calculates bidirectional alignment matching score based on key image patch set and redundant image patch set; obtains fine-grained semantic alignment loss based on bidirectional alignment matching score; The retrieval module constructs a retrieval model based on global alignment loss, self-supervised semantic reconstruction loss, and fine-grained semantic alignment loss. Iterative optimization of the retrieval model is performed until the convergence condition is met, and the final cross-modal retrieval results are output.

[0038] This invention combines coarse-grained global contrastive learning with fine-grained interaction. Global alignment ensures overall semantic consistency, while fine-grained alignment captures rich local details. The two complement each other through joint optimization, effectively solving the problem that single-granularity alignment is prone to getting trapped in local optima or semantic bias, and significantly improving the accuracy of cross-modal retrieval. Attached Figure Description

[0039] Figure 1 This is a flowchart of a cross-modal retrieval method based on image patch selection and dual granularity alignment in one embodiment; Figure 2 This is a block diagram of a cross-modal retrieval system based on image patch selection and dual granularity alignment in one embodiment; Figure 3 This is a comparative experimental result graph from one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] Example 1: This embodiment provides, as follows: Figure 1 The cross-modal retrieval method based on image patch selection and dual granularity alignment shown includes: S1: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input the image data into the pre-trained visual encoder to obtain the visual image patch feature sequence, and input the text data into the pre-trained text encoder to obtain the text word feature sequence. S2: Map the visual image patch feature sequence and the text word feature sequence and perform global pooling to obtain the global image feature and the global text feature. Then perform coarse-grained global semantic alignment to obtain the global alignment loss. S3: Calculate the self-supervised semantic reconstruction loss based on the visual image patch feature sequence, select the key image patch set, and generate the redundant image patch set. S4: Calculate the bidirectional alignment matching score based on the key image patch set and the redundant image patch set; obtain the fine-grained semantic alignment loss based on the bidirectional alignment matching score; S5.: Construct a retrieval model based on global alignment loss, self-supervised semantic reconstruction loss, and fine-grained semantic alignment loss. Iteratively optimize the retrieval model until the convergence condition is met, and output the final cross-modal retrieval results.

[0041] This embodiment combines coarse-grained global contrastive learning with fine-grained interaction. Global alignment ensures overall semantic consistency, while fine-grained alignment captures rich local details. The two complement each other through joint optimization, effectively solving the problem that single-granularity alignment is prone to getting trapped in local optima or semantic bias, and significantly improving the accuracy of cross-modal retrieval.

[0042] Example 2: This embodiment further discloses information based on Embodiment 1: Further, in step S1, the image data is input into a pre-trained visual encoder to obtain a visual image patch feature sequence. Specifically, the Vision Transformer is used as a pre-trained visual encoder to segment the image data into... From a fixed-size image patch, visual image patch feature sequences are extracted. ,in As category morphemes, For feature dimensions.

[0043] Further, in step S1, the text data is input into a pre-trained text encoder to obtain a text word feature sequence. Specifically, BERT is used as the text encoder to convert the text data into a word sequence, and the text word feature sequence is extracted. ,in This represents the text length.

[0044] Further, in step S2, the mapping and global pooling of the visual image patch feature sequence and text word feature sequence to obtain the global image features and global text features are specifically as follows: Construct a learnable linear projection layer and Through learnable linear projection layers and Map visual and textual features to a dimension of The common embedding space is then used to perform global pooling to obtain global image features and global text features.

[0045] Further, in step S2, coarse-grained global semantic alignment is performed, and the global alignment loss is obtained as follows: Minimize global contrast loss The calculation formula is as follows:

[0046] in, The image-to-text contrast loss is calculated using the following formula:

[0047] In the formula, For batch size, Indicates the first Global features of an image With the Text global features Cosine similarity between them Temperature coefficient; The text-to-image contrast loss is calculated in the same way as... correspond.

[0048] Furthermore, in step S3, specifically: S3.1: Utilizing visual image patch feature sequences With global features of the image Calculate the intramodal self-attention score To capture the saliency within the visual image; simultaneously, utilizing the feature sequences of visual image patches. With text global features Calculate cross-modal attention scores This is to introduce text semantic supervision; the specific calculation formula is as follows:

[0049]

[0050] In the formula, Indicates the scaling factor. This represents the normalization function, specifically using the Softmax function to map scores to the interval [0, 1]. Its definition is... Finally, the weighted fusion calculation is performed on the first... The final comprehensive saliency score of each image patch :

[0051] in, To balance the hyperparameters of intramodal attention and crossmodal attention; S3.2: The variance of the feature sequences of all visual image patches is used to measure the dispersion of the image semantic distribution. The calculation formula is as follows:

[0052] Higher variance indicates a wider semantic distribution, requiring the retention of more image patches; S3.3 Calculate the attention concentration of the image. The degree of attention focus is measured using the entropy of the comprehensive saliency score distribution. The calculation formula is as follows:

[0053] in To The result of Softmax normalization; higher entropy indicates distracted attention and uncertainty, requiring the retention of more image patches; S3.4 Calculate the dynamic retention rate Based on the semantic discreteness and attention concentration Adaptive calculation of image dynamic retention rate :

[0054] in, and These are the set lower and upper limits for retention rates, respectively. To balance the coefficients of the two complexity metrics, and These represent the semantic dispersion and attention concentration after batch normalization, respectively. S3.5 Dynamically Select Key Image Patches: Based on the calculated dynamic retention rate Total number of image blocks Determine the number to retain. Based on the comprehensive significance score calculated in step S3.1 Sort all image patches and select the highest-scoring ones. The key image patch set consists of several image patches. The remaining image blocks form a redundant image block set. ; S3.6: To ensure that the adaptively preserved image patches maintain the semantic integrity of the original image, a self-supervised semantic reconstruction loss is constructed. .

[0055] Further, in step S3.6, specifically: Self-supervised semantic reconstruction loss ; Calculate the global average pooling representation of the feature sequences of all visual image patches in the original image. ; Calculate the set of key image patches Global average pooling representation of feature sequences of all visual image patches in the dataset. The calculation formula is:

[0056] Calculate the mean squared error (MSE) between the two as the semantic reconstruction loss:

[0057] This loss function is used to constrain the dynamic selection process and prevent the loss of key semantic information.

[0058] Further, in step S4, specifically: S4.1 Feature Aggregation and Compensation: Defining a Set of Key Image Patches Set the aggregation ratio Number of target aggregation features The feature sequences of key visual image patches are mapped using an aggregation network composed of multilayer perceptrons, and the aggregation weight matrix is ​​calculated using the Softmax function. The calculation formula is:

[0059] Secondly, based on the weight matrix Weighted aggregation of key image patches yields Discriminative features :

[0060] Simultaneously, a context-aware fusion strategy is adopted, based on the set of redundant image patches. Calculate attention weights based on feature magnitudes to generate compensated features. :

[0061] Ultimately constitutes an enhanced visual representation ; S4.2 Fine-grained image patch-word alignment: Computationally enhancing visual representations Text features Image patch-Word similarity matrix between ,in The matching score is calculated using a bidirectional maximum correspondence mechanism.

[0062] Final two-way aligned total score .

[0063] Fine-grained semantic alignment loss The loss function is calculated using a two-way triplet, and the specific formula is as follows:

[0064]

[0065]

[0066] In the formula, For boundary parameters, Positive sample pairs The two-way alignment matching score; Indicates the image within the current batch. The most similar hard-to-bear sample text, Match a score to it; Indicates the text within the current batch The most similar hard-to-bear sample image. Match a score to it.

[0067] Further, in step S5, specifically: The overall objective function of the retrieval model It integrates global alignment loss, semantic reconstruction loss, and fine-grained semantic alignment loss, with the specific formulas defined as follows:

[0068] in: The global alignment loss is calculated based on the global feature similarity between the image and the text. This is the self-supervised semantic reconstruction loss, used to constrain semantic consistency during the dynamic selection process; This is the fine-grained semantic alignment loss.

[0069] This embodiment combines coarse-grained global contrastive learning with fine-grained interaction. Global alignment ensures overall semantic consistency, while fine-grained alignment captures rich local details. The two complement each other through joint optimization, effectively solving the problem that single-granularity alignment is prone to getting trapped in local optima or semantic bias, and significantly improving the accuracy of cross-modal retrieval.

[0070] Example 3: This embodiment provides, as follows: Figure 2 The cross-modal retrieval system based on image patch selection and dual granularity alignment shown includes: Data acquisition and feature extraction module: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input image data into the pre-trained visual encoder to obtain visual image patch feature sequence, and input text data into the pre-trained text encoder to obtain text word feature sequence; Coarse-grained alignment module: Mapping and global pooling are performed on the visual image block feature sequence and the text word feature sequence to obtain global image features and global text features. Then, coarse-grained global semantic alignment is performed to obtain global alignment loss. Self-supervised semantic reconstruction loss module: Calculates self-supervised semantic reconstruction loss based on the feature sequence of visual image patches, selects the key image patch set, and generates a redundant image patch set. Fine-grained semantic alignment module: Calculates bidirectional alignment matching score based on key image patch set and redundant image patch set; obtains fine-grained semantic alignment loss based on bidirectional alignment matching score; The retrieval module constructs a retrieval model based on global alignment loss, self-supervised semantic reconstruction loss, and fine-grained semantic alignment loss. Iterative optimization of the retrieval model is performed until the convergence condition is met, and the final cross-modal retrieval results are output.

[0071] This embodiment combines coarse-grained global contrastive learning with fine-grained interaction. Global alignment ensures overall semantic consistency, while fine-grained alignment captures rich local details. The two complement each other through joint optimization, effectively solving the problem that single-granularity alignment is prone to getting trapped in local optima or semantic bias, and significantly improving the accuracy of cross-modal retrieval.

[0072] Example 4: This embodiment provides a cross-modal retrieval method based on image patch selection and dual-granularity alignment, including: S1, Multimodal Feature Extraction In this embodiment, the input data is an image-text pair. For the visual modality, a pre-trained VisionTransformer (e.g., ViT-base) is used as the visual encoder. The input image... Adjusted to Resolution, and divided into The size is Fixed patches are generated. These patches, after linear projection and positional encoding, are input into the Transformer layer to extract visual feature sequences. ,in For category tokens, This represents the original feature dimension. For the text modality, a pre-trained BERT (e.g., BERT-base) is used as the text encoder. The input text... The tokens are converted into a token sequence (containing [CLS] and [SEP] markers) by a tokenizer, and then input into the BERT model to extract the text feature sequence. ,in The text length is used. Finally, it is processed through two independent learnable linear projection layers. and Unifying the mapping of visual and textual features to a dimension of A public, interconnected, embedded space.

[0073] S2, Global Semantic Alignment To establish coarse-grained cross-modal associations, the visual patch feature sequence and text token feature sequence obtained in step S1 are subjected to global average pooling to obtain a global visual vector. and global text vector The global feature representation is optimized using the contrastive learning loss function (InfoNCE Loss). Specifically, for a batch of... For each image-text pair, calculate the global contrast loss. :

[0074] in and Representing the retrieval losses for image-to-text and text-to-image respectively, with temperature coefficients... Set to 0.07. Alignment of the global semantic space is achieved by maximizing the cosine similarity of matching pairs and minimizing the similarity of unmatched pairs.

[0075] S3, Dynamic Sparse Selection This is the core step of the invention, designed to adaptively retain key patches based on the complexity of the image content.

[0076] S3.1 Comprehensive saliency score of computer vision patch First, calculate the intramodal self-attention score. and cross-modal attention score :

[0077] Then, the final saliency score for each patch is calculated through weighted fusion. :

[0078] In this embodiment, the balancing hyperparameters It is set to 0.5 to balance visual salience and textual semantic relevance.

[0079] S3.2 Calculate the dynamic retention rate To quantify the complexity of an image, two metrics are calculated: Semantic Discreteness The variance of all patch features is used to calculate the degree of dispersion of the image semantic distribution.

[0080] Attention Concentration Using saliency scoring The entropy calculation of the distribution reflects the degree of attention focus.

[0081] These two metrics are normalized within a batch to obtain... and The dynamic retention rate of the image is adaptively calculated using the following formula. :

[0082] In this embodiment, a lower limit for retention rate is set. upper limit Balance coefficient This means that for images with complex backgrounds and scattered semantics, the model will automatically calculate a higher retention rate to preserve more details; while for images with prominent subjects and simple backgrounds, it will calculate a lower retention rate to remove redundancy.

[0083] S3.3 Dynamic Filtering and Semantic Reconstruction Constraints According to the calculation Keep rating highest Each patch constitutes a key patch set. To prevent the loss of key semantic information during dynamic filtering, a self-supervised semantic reconstruction loss function is constructed. :

[0084] in It is the average pooling feature of the original full patch. It is the average pooling feature of the key patches after filtering. By minimizing this mean squared error, the model is forced to select those patches that best represent the semantics of the entire graph.

[0085] S4, Two-Granularity Feature Aggregation and Alignment S4.1 Feature Aggregation and Compensation In this embodiment, in order to reduce computational redundancy while preserving key semantics, an aggregation ratio parameter is set. The specific implementation steps are as follows: 1. Key Feature Aggregation: First, determine the number of features after aggregation. A convergence network incorporating a multilayer perceptron (MLP) is used to analyze the key patch set. Process it. Specifically, The input to the MLP layer is used for feature mapping, and then normalized across the $k$-th patch dimensions using the Softmax function to generate a single dimension. Aggregated weight matrix :

[0086] Subsequently, using this weight matrix We perform a weighted summation of the key patch features to obtain A compact and discriminative aggregate feature (in ):

[0087] This step effectively captures the semantic dependencies between patches, compressing scattered visual information into a more compact representation.

[0088] 2. Redundant Feature Compensation: To further salvage information discarded during the filtering process, a context-aware fusion strategy is employed. This applies to the discarded patch set. Calculate the exponentially normalized weights of its feature magnitudes, and then weight and fuse all discarded patches into a single global compensation feature. :

[0089] 3. Constructing the final representation: The final enhanced visual representation Composed of category tokens, aggregated k' key features, and global compensation features It is pieced together, that is This representation will be used for subsequent fine-grained interactions with text features.

[0090] S4.2 Fine-grained Patch-Word Alignment Computational augmented visual representation Text features The cosine similarity matrix between them. A bidirectional maximum correspondence mechanism is used to calculate the fine-grained matching score: that is, for each visual feature, find the best-matching text token, and for each text token, find the best-matching visual feature, then calculate the average to obtain the result. and Finally, the summation yields the fine-grained aligned total score.

[0091] S5, Model Training and Output Construct the overall objective function Perform end-to-end training:

[0092] in A two-way triplet loss function is used, with boundary parameters... Set to 0.2.

[0093] Optimization was performed using the AdamW optimizer, with an initial learning rate set to... The number of training epochs is 35.

[0094] To verify the effectiveness of the proposed cross-modal retrieval method based on dynamic patch selection and dual granularity alignment (DPSDGA) in this embodiment, comprehensive experimental verification was conducted on two widely used benchmark datasets, Flickr30K and MS-COCO.

[0095] 1. Experimental Setup All experiments were conducted in a unified hardware and software environment to ensure fairness.

[0096] Hardware environment: The server is equipped with an NVIDIA GeForce RTX 4090 GPU (24GB VRAM).

[0097] Software environment: Operating system is Ubuntu, deep learning framework is PyTorch, CUDA version is 12.4.

[0098] Parameter settings: Optimize using the AdamW optimizer, with an initial learning rate of 2e-4, a training epoch count of 35, and a batch size of 128.

[0099] 2. Comparison of experimental results To demonstrate the superiority of the method in this invention, mainstream cross-modal retrieval methods were selected as benchmarks for comparison, including VSE++, SCAN, SGR, CHAN, and the latest LAPS method. Recall@K (K=1, 5, 10) was used as the evaluation metric, and rSum was the sum of all recall metrics.

[0100] (1) Experimental results of the Flickr30K dataset As shown in Table 1, when using ViT-base as the visual backbone network, the method of this invention outperforms the comparative methods in all metrics. In particular, for the most crucial R@1 metric, image-to-text retrieval reaches 75.6%, text-to-image retrieval reaches 62.7%, and the overall recall (rSum) reaches 511.8%, representing a 6.0% improvement over the state-of-the-art LAPS method. When switching to the stronger Swin-Transformer backbone, rSum further increases to 539.8%.

[0101]

[0102] (2) Experimental results of MS-COCO dataset As shown in Table 2, this invention maintains its performance advantage on the larger MS-COCO dataset. Under the ViT-base architecture, rSum reaches 523.7%; under the Swin-base architecture, rSum reaches 541.1. This demonstrates that the dynamic sparse selection strategy proposed in this invention has good generalization ability on large-scale data.

[0103]

[0104] 3. Validity analysis of dynamic patch selection, such as... Figure 3 As shown in the figure, experimental statistics show that the dynamic retention rate calculated by the model is... It is significantly positively correlated with image complexity. Figure 3 The horizontal axis represents the normalized image complexity score, and the vertical axis represents the patch retention rate predicted by the model. It can be seen that the data point distribution shows a clear upward trend. This means that in complex scenes, the model automatically retains more patches to capture details; in simple scenes, the model significantly reduces the number of patches to filter out noise. This mechanism of adaptively allocating computational resources based on content complexity is the key source of the performance improvement in this invention.

Claims

1. A cross-modal retrieval method based on image patch selection and dual-granularity alignment, characterized in that, include: S1: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input the image data into the pre-trained visual encoder to obtain the visual image patch feature sequence, and input the text data into the pre-trained text encoder to obtain the text word feature sequence. S2: Map the visual image patch feature sequence and the text word feature sequence and perform global pooling to obtain the global image feature and the global text feature. Then perform coarse-grained global semantic alignment to obtain the global alignment loss. S3: Calculate the self-supervised semantic reconstruction loss based on the visual image patch feature sequence, select the key image patch set, and generate the redundant image patch set. S4: Calculate the bidirectional alignment matching score based on the key image patch set and the redundant image patch set; obtain the fine-grained semantic alignment loss based on the bidirectional alignment matching score; S5: Construct a total objective function that includes global alignment loss, semantic reconstruction loss, and semantic alignment loss. Iteratively optimize the model parameters through backpropagation until the convergence condition is met, and output the final cross-modal retrieval results.

2. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S1, the image data is input into a pre-trained visual encoder to obtain a visual image patch feature sequence. Specifically, the Vision Transformer is used as a pre-trained visual encoder to segment the image data into... From a fixed-size image patch, visual image patch feature sequences are extracted. ,in As category morphemes, For feature dimensions.

3. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S1, the text data is input into a pre-trained text encoder to obtain a text word feature sequence. Specifically, BERT is used as the text encoder to convert the text data into a word sequence and extract the text word feature sequence. ,in This represents the text length.

4. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S2, the image global features and text global features are obtained by mapping and global pooling based on the visual image patch feature sequence and text word feature sequence, specifically as follows: Construct a learnable linear projection layer and Through learnable linear projection layers and Map visual and textual features to a dimension of The common embedding space is then used to perform global pooling to obtain global image features and global text features.

5. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S2, coarse-grained global semantic alignment is performed, and the global alignment loss is obtained as follows: Minimize global contrast loss The calculation formula is as follows: in, The image-to-text contrast loss is calculated using the following formula: In the formula, For batch size, Indicates the first Global features of an image With the Text global features Cosine similarity between them Temperature coefficient; The text-to-image contrast loss is calculated in the same way as... correspond.

6. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S3, specifically: S3.1: Utilizing visual image patch feature sequences With global features of the image Calculate the intramodal self-attention score To capture the saliency within the visual image; simultaneously, utilizing the feature sequences of visual image patches. With text global features Calculate cross-modal attention scores This is to introduce text semantic supervision; the specific calculation formula is as follows: In the formula, Indicates the scaling factor. This represents the normalization function, specifically using the Softmax function to map scores to the interval [0, 1]. Its definition is... Finally, the weighted fusion calculation is performed on the first... The final comprehensive saliency score of each image patch : in, To balance the hyperparameters of intramodal attention and crossmodal attention; S3.2: The variance of the feature sequences of all visual image patches is used to measure the dispersion of the image semantic distribution. The calculation formula is as follows: Higher variance indicates a wider semantic distribution, requiring the retention of more image patches; S3.3 Calculate the attention concentration of the image. The degree of attention focus is measured using the entropy of the comprehensive saliency score distribution. The calculation formula is as follows: in To The result of Softmax normalization; higher entropy indicates distracted attention and uncertainty, requiring the retention of more image patches; S3.4 Calculate the dynamic retention rate Based on the semantic discreteness and attention concentration Adaptive calculation of image dynamic retention rate : in, and These are the set lower and upper limits for retention rates, respectively. To balance the coefficients of the two complexity metrics, and These represent the semantic dispersion and attention concentration after batch normalization, respectively. S3.5 Dynamically Select Key Image Patches: Based on the calculated dynamic retention rate Total number of image blocks Determine the number to retain. Based on the comprehensive significance score calculated in step S3.1 Sort all image patches and select the highest-scoring ones. The key image patch set consists of several image patches. The remaining image blocks form a redundant image block set. ; S3.6: To ensure that the adaptively preserved image patches maintain the semantic integrity of the original image, a self-supervised semantic reconstruction loss is constructed. .

7. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S3.6, specifically: Self-supervised semantic reconstruction loss ; Calculate the global average pooling representation of the feature sequences of all visual image patches in the original image. ; Calculate the set of key image patches Global average pooling representation of feature sequences of all visual image patches in the dataset. The calculation formula is: Calculate the mean squared error (MSE) between the two as the semantic reconstruction loss: This loss function is used to constrain the dynamic selection process and prevent the loss of key semantic information.

8. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S4, specifically: S4.1 Feature Aggregation and Compensation: Defining a Set of Key Image Patches Set the aggregation ratio Number of target aggregation features The feature sequences of key visual image patches are mapped using an aggregation network composed of multilayer perceptrons, and the aggregation weight matrix is ​​calculated using the Softmax function. The calculation formula is: Secondly, based on the weight matrix Weighted aggregation of key image patches yields Discriminative features : Simultaneously, a context-aware fusion strategy is adopted, based on the set of redundant image patches. Calculate attention weights based on feature magnitudes to generate compensated features. : Ultimately constitutes an enhanced visual representation ; S4.2 Fine-grained image patch-word alignment: Computationally enhancing visual representations Text features Image patch-Word similarity matrix between ,in The matching score is calculated using a bidirectional maximum correspondence mechanism. Final two-way aligned total score ; Fine-grained semantic alignment loss The loss function is calculated using a two-way triplet, and the specific formula is as follows: In the formula, For boundary parameters, Positive sample pairs The two-way alignment matching score; Indicates the image within the current batch. The most similar hard-to-bear sample text, Match a score to it; Indicates the text within the current batch The most similar hard-to-bear sample image. Match a score to it.

9. The cross-modal retrieval method based on image patch selection and dual-granularity alignment according to claim 1, characterized in that, In step S5, specifically: The overall objective function of the retrieval model It integrates global alignment loss, semantic reconstruction loss, and fine-grained semantic alignment loss, with the specific formulas defined as follows: in: The global alignment loss is calculated based on the global feature similarity between the image and the text. This is the self-supervised semantic reconstruction loss, used to constrain semantic consistency during the dynamic selection process; This is the fine-grained semantic alignment loss.

10. A cross-modal retrieval system based on image patch selection and dual-granularity alignment, characterized in that, include: Data acquisition and feature extraction module: Acquire image data and text data, construct a pre-trained visual encoder and a pre-trained text encoder, input image data into the pre-trained visual encoder to obtain visual image patch feature sequence, and input text data into the pre-trained text encoder to obtain text word feature sequence; Coarse-grained alignment module: Mapping and global pooling are performed on the visual image block feature sequence and the text word feature sequence to obtain global image features and global text features. Then, coarse-grained global semantic alignment is performed to obtain global alignment loss. Self-supervised semantic reconstruction loss module: Calculates self-supervised semantic reconstruction loss based on the feature sequence of visual image patches, selects the key image patch set, and generates a redundant image patch set. Fine-grained semantic alignment module: Calculates bidirectional alignment matching score based on key image patch set and redundant image patch set; obtains fine-grained semantic alignment loss based on bidirectional alignment matching score; The retrieval module constructs a retrieval model based on global alignment loss, self-supervised semantic reconstruction loss, and fine-grained semantic alignment loss. Iterative optimization of the retrieval model is performed until the convergence condition is met, and the final cross-modal retrieval results are output.

Images