Image-text retrieval method for multi-view semantic topology alignment based on VLMs

Abstract

The invention relates to the technical field of cross-modal retrieval, in particular to a VLMs-based image-text retrieval method for multi-view semantic topology alignment, which comprises the following steps: acquiring a cross-modal image-text pair, and expanding text description through a multi-modal large model to construct double-granularity semantic features containing global anchor points and local primitives; image visual features are extracted and projected to a plurality of semantic subspaces, a semantic prototype is constructed based on local primitives, the visual features and the prototype are interacted, global anchor points are fused, and a multi-view visual embedding set is generated; aggregating the set to generate visual centroids, pushing the visual centroids into a centroid pool to calculate global discriminant losses, and calculating dense and maximum alignment losses for the set; and finally, combining the three losses to construct a target function training network to obtain a retrieval model. According to the method, the complexity and ambiguity of visual contents are effectively decoupled, and accurate mapping and alignment of image-text features are realized.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and in particular to an image-text retrieval method based on VLMs with multi-view semantic topology alignment. Background Technology

[0002] With the explosive growth of multimedia data, cross-modal image-text retrieval plays a crucial role in practical applications such as search engines and cross-media content understanding. The core of cross-modal retrieval lies in how to accurately measure the semantic similarity between different modalities of data within a unified feature space.

[0003] However, traditional cross-modal retrieval mostly uses a single vector mapping, which is difficult to decouple from complex visual polysemous features, resulting in problems such as local detail matching deviations in images and text and low retrieval accuracy. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides an image-text retrieval method based on VLMs with multi-view semantic topology alignment. It aims to improve the problem that traditional cross-modal retrieval mostly uses a single vector mapping, which is difficult to decouple from complex visual polysemous features, resulting in local detail matching deviations in images and text and low retrieval accuracy.

[0005] This invention provides the following technical solution: an image-text retrieval method based on VLMs with multi-view semantic topology alignment, comprising the following steps: S1. Obtain image-text pairs from the cross-modal dataset; expand the original text of the image-text pairs using a multimodal large language model, extract global semantic anchors and local descriptive primitives, and construct dual-granularity semantic features; S2. Extract the visual features of the original image in the image-text pair, and project the visual features into multiple semantic subspaces; S3. Within each of the semantic subspaces, construct a semantic prototype based on the local descriptive primitives, perform multi-head cross-attention interaction between the visual features and the semantic prototypes, fuse the global semantic anchors, and generate a multi-view visual embedding set. S4. Aggregate the multi-view visual embedding set to generate visual centroids, push the visual centroids into the cross-view centroid pool, and calculate the global discrimination loss based on the centroid features and text modal features in the cross-view centroid pool; S5. For the multi-view visual embedding set, calculate the dense alignment loss and the maximum alignment loss to perform multi-view alignment; S6. Construct an objective function by combining the global discrimination loss, the dense alignment loss, and the maximize alignment loss. Train the retrieval network based on the objective function to obtain an image-text retrieval model.

[0006] By adopting the above technical solution, dual-granularity semantic features are constructed and multi-view interaction is performed in multiple semantic subspaces, thereby achieving accurate alignment of images and text. This improves the problem that traditional cross-modal retrieval mostly uses single vector mapping, which is difficult to decouple from complex visual polysemous features, resulting in local detail matching deviations and low retrieval accuracy.

[0007] Optionally, in S1, the step of expanding the original text of the image-text pair using a multimodal large language model, extracting global semantic anchors and local descriptive primitives, and constructing dual-granularity semantic features includes: The original text is expanded and described using a multimodal large language model with frozen network parameters. The global high-dimensional features of the extended description text are extracted by a text encoder and used as the global semantic anchor. Fine-grained sequence features of the extended description text are extracted as the local description primitives; The global semantic anchor and the local descriptive primitive are combined to form the dual-granularity semantic feature.

[0008] Optionally, in S2, extracting the visual features of the original image in the image-text pair and projecting the visual features into multiple semantic subspaces includes: The original image is processed by feature encoding through a visual backbone network to obtain discrete visual block feature sequences. A multilayer perceptron network and a linear mapping layer corresponding to a specific semantic subspace are used to perform nonlinear projection processing on the visual block feature sequence to obtain an initial visual embedding specific to each semantic subspace.

[0009] Optionally, in S3, the step of constructing a semantic prototype based on the local descriptive primitives and performing multi-head cross-attention interaction between the visual features and the semantic prototype includes: The semantic prototype is obtained by performing attention aggregation operation on the local description primitives as query keys and values. Within each of the semantic subspaces, the initial visual embedding is used as the query vector, and the semantic prototype is used as the key vector and value vector to perform multi-head cross-attention operation to obtain the visual features after interaction.

[0010] Optionally, in S3, fusing the global semantic anchors to generate a multi-view visual embedding set includes: Max pooling is performed on the visual features after interaction using a scaling factor, while average pooling is performed on the initial visual embedding. The features processed by max pooling and the features processed by average pooling are weighted by residual fusion to obtain aggregated visual features. The aggregated visual features and the global semantic anchors are combined and normalized to generate the final visual embedding of a specific semantic subspace. The final visual embeddings of all the specific semantic subspaces are combined into the multi-view visual embedding set.

[0011] Optionally, in S4, aggregating the multi-view visual embedding set to generate a visual centroid, and pushing the visual centroid into the cross-view centroid pool includes: The visual centroid is generated by summing and averaging all view features in the multi-view visual embedding set. An exponential moving average strategy is used to update the network parameters of the key visual encoders within the retrieval network. The visual centroid is stored in the cross-view centroid pool maintained by the key visual encoder.

[0012] Optionally, in S4, calculating the global discrimination loss based on the centroid features and text modality features in the cross-view centroid pool includes: The positive sample similarity between the centroid features in the cross-view centroid pool and the matched key text features is calculated based on the cosine similarity function. The negative sample similarity between the centroid features in the cross-view centroid pool and the negative sample features in the text sample pool is calculated based on the cosine similarity function. The global discriminant loss from image to text and the global discriminant loss from text to image are calculated based on the positive sample similarity and the negative sample similarity. The global discrimination loss is obtained by adding the global discrimination loss from image to text to the global discrimination loss from text to image.

[0013] Optionally, in S5, the calculation of dense alignment loss includes: Extract the most difficult negative sample text from the training batch based on the global maximum similarity matching criterion; The dense alignment loss is calculated based on the similarity between the final visual embedding of the specific semantic subspace and the matching key text features, the similarity between the final visual embedding of the specific semantic subspace and the most difficult negative sample text, and the interval boundary value.

[0014] Optionally, in S5, the calculation of maximizing the alignment loss includes: Calculate the similarity between all view features and key text features in the multi-view visual embedding set; Filter out the maximum similarity response value from all the stated similarities; The maximum alignment loss is constructed based on the maximum similarity response value.

[0015] Optionally, in S6, a target function is constructed by combining the global discrimination loss, the dense alignment loss, and the maximized alignment loss. The retrieval network is then trained based on this target function to obtain an image-text retrieval model, including: Hyperparameter weights are assigned to the global discrimination loss, the dense alignment loss, and the maximized alignment loss, respectively. The objective function is obtained by linearly summing the global discrimination loss, the dense alignment loss, and the maximized alignment loss after assigning the hyperparameter weights. Based on the objective function, the backpropagation algorithm is used to update the network parameters of the retrieval network until the retrieval network reaches a convergent state to obtain the image-text retrieval model.

[0016] The present invention has the following beneficial effects: 1. In this invention, by constructing dual-granularity semantic features and performing multi-view interaction in multiple semantic subspaces, precise alignment of images and text is achieved. This improves upon the problem that traditional cross-modal retrieval mostly uses a single vector mapping, which is difficult to decouple from complex visual polysemous features, resulting in local detail matching deviations and low retrieval accuracy.

[0017] 2. In this invention, an extended description is generated by a multimodal large language model and dual-granularity features are extracted to construct a semantic prototype, thereby guiding the directional activation of visual features. This improves the problem that traditional feature extraction mostly uses direct encoding of the original text, which is difficult to align across modal semantic spaces due to the brevity or semantic sparsity of the original description text.

[0018] 3. In this invention, visual centroids are generated by aggregating multi-view features and pushed into the cross-view centroid pool to construct a global loss, thereby expanding the stable negative sample capacity. This improves the problem that traditional negative sample construction mostly adopts direct comparison within batches, which, due to limited batch capacity and sample distribution, causes the network model to easily fall into local optima.

[0019] 4. In this invention, multi-view alignment optimization is performed by calculating dense alignment loss and maximizing alignment loss, thereby forcing each subspace to capture complementary details. This improves the problem of gradient sparsity in backpropagation caused by the lack of effective mining of specific hard negative samples, which is mostly based on a single global contrast value. Attached Figure Description

[0020] Figure 1 This is a flowchart of the image-text retrieval method based on VLMs with multi-view semantic topology alignment proposed in this invention; Figure 2This is a flowchart of the dual-granularity semantic feature construction process of the image-text retrieval method based on VLMs with multi-view semantic topology alignment proposed in this invention. Figure 3 This is a flowchart illustrating the objective function construction and network training process of the image-text retrieval method based on VLMs with multi-view semantic topology alignment proposed in this invention. Figure 4 This is a comparison chart of the performance and inference speed of the image-text retrieval model of the proposed image-text retrieval method based on VLMs and existing technologies. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1: In a first embodiment of the present invention, the present invention provides an image-text retrieval method based on VLMs with multi-view semantic topology alignment, such as... Figures 1-3 As shown, it includes the following steps: S1. Obtain image-text pairs from the cross-modal dataset; expand the original text of the image-text pairs using a multimodal large language model, extract global semantic anchors and local descriptive primitives, and construct dual-granularity semantic features. Furthermore, in S1, the original text of the image-text pair is expanded and described using a multimodal large language model, extracting global semantic anchors and local descriptive primitives to construct dual-granularity semantic features, including: Expanded descriptive text is output from the original text using a multimodal large language model with frozen network parameters. The text encoder extracts global high-dimensional features of the extended descriptive text as global semantic anchors; Fine-grained sequence features of the extended descriptive text are extracted as local descriptive primitives; Global semantic anchors and local descriptive primitives are combined into dual-granularity semantic features.

[0023] Specifically, in step S1, image-text pairs are obtained from the cross-modal dataset; the original text of the image-text pairs is expanded and described using a multimodal large language model, global semantic anchors and local descriptive primitives are extracted, and dual-granularity semantic features are constructed.

[0024] This step aims to address the asymmetry of modal information in image-text alignment within raw cross-modal data. Since images typically contain global and complex visual entity interactions, while manually annotated raw text descriptions often possess subjective locality, directly aligning the two can easily lead to semantic loss within the feature space. An external multimodal large language model is introduced to generate extended descriptive text containing contextual knowledge, providing explicit feature priors for subsequent view-specific projections and fine-grained interactions of visual features.

[0025] The acquired original image and text, along with pre-defined prompts, are used as input data and fed into a multimodal large language model with its internal network parameters frozen for inference, outputting expanded descriptive text. The corresponding calculation formula for this process is: ; in This represents the extended descriptive text output by the multimodal large language model; This represents the raw image data input to a multimodal large language model; This represents the original text data that is matched with and input together with the original image data; This indicates the set prompt command parameters; This indicates a multimodal large language model that is invoked and whose network parameters are completely frozen.

[0026] After obtaining the extended description text, feature encoding processing is performed from both global and local dimensions.

[0027] For the global dimension, a pre-trained text encoder is used to extract high-dimensional global features that expand the descriptive text, which are then used as global semantic anchors. The corresponding calculation formula is: ; in, This represents the extracted global semantic anchor points, used to characterize cross-modal global semantic information at a macroscopic level; This represents a pre-trained text encoder; Extended descriptive text representing the output of the preceding operations; This indicates that it belongs to the symbol category, and the extracted global semantic anchor points are defined by the dimension. The real-valued feature space; Indicates the dimension size as The real-valued feature space; The feature vector dimension value representing the global semantic anchor point.

[0028] For the local dimension, the embedding layer processing function within the text backbone network is used to extract the sequence features of the extended descriptive text, which are then used as local descriptive primitives. The corresponding calculation formula is: ; in The extracted local descriptive primitives are represented as discrete, fine-grained sequence features. This represents the embedding layer processing function of the text backbone network; Extended descriptive text representing the output of the preceding operations; The symbol represents the element belonging to the extracted local descriptive primitive, and the primitive is defined as belonging to a size of [size missing]. The space of real characteristic matrices; Indicates size is The space of real characteristic matrices; The numerical value representing the sequence length of the extracted local descriptive primitives; This represents the feature dimension values ​​of each independent sequence unit in the local descriptive primitive.

[0029] After extracting data from the above dimensions, the obtained global semantic anchors and local descriptive primitives are combined and merged to construct dual-granularity semantic features. The generated dual-granularity semantic features are then output to the visual processing branch in the cross-modal network, serving as query conditions for subsequent partitioning of the visual semantic subspace and execution of multi-view cross-attention interactions.

[0030] S2. Extract the visual features of the original image in the image-text pair and project the visual features into multiple semantic subspaces; Furthermore, in S2, visual features of the original image in the image-text pair are extracted, and these visual features are projected into multiple semantic subspaces, including: The original image is processed by feature encoding through a visual backbone network to obtain discrete visual block feature sequences. A multilayer perceptron network and a linear mapping layer corresponding to a specific semantic subspace are used to perform nonlinear projection processing on the visual block feature sequence to obtain the initial visual embedding specific to each semantic subspace.

[0031] Specifically, the core objective of this step is to overcome the limitations of traditional retrieval models that rely on a single visual vector for representation. Original images typically intertwine complex object shapes and multi-perspective action interactions; forcibly compressing them into a single static feature inevitably leads to the loss of local details. By extracting discrete visual blocks and using a neural network to distribute them across multiple independent semantic subspaces, the intertwined visual content can be initially decomposed and separated along the feature dimension. This provides underlying data support for subsequent networks to utilize prior textual data for targeted retrieval and fusion of specific visual features.

[0032] In the feature extraction stage, the original image from the image-text pair is used as input data and directly fed into the pre-trained visual backbone network for feature encoding. To preserve fine-grained spatial details within the image, the feature map output from the last layer of the visual backbone network is extracted and completely flattened in the spatial dimension, ultimately outputting a discrete sequence of visual block features. The mathematical formula corresponding to this operation is: ; in This represents the visual block feature sequence output after flattening. This indicates the feature mapping function of the pre-trained visual backbone network used. This indicates the input to the visual backbone network. One original image sample. This represents the first global visual feature unit in the feature sequence responsible for capturing overall image information. to These represent the visual block feature units that carry local details in the feature sequence, in sequence. This represents the total number of visual blocks after the input image has been segmented. Mathematics belongs to the symbolic domain, defining the membership space of the output feature matrix. Indicates size is A space of real-valued feature matrices of dimension 1. This represents the specific dimensional value of the output feature of a single layer in the visual backbone network.

[0033] After obtaining the aforementioned visual block feature sequence, the nonlinear projection processing stage begins. The visual block feature sequence is used as input data and fed into a pre-defined projection network module. For multiple semantic subspaces pre-defined by the model, the projection network module invokes a multilayer perceptron network with shared parameters, and combines it with linear mapping layers corresponding one-to-one to each specific semantic subspace to perform superposition calculations on the input sequence data, outputting initial visual embeddings specific to each semantic subspace. For the first... The specific projection mapping formula for a particular semantic subspace is as follows: ; in This indicates that the output after overlay mapping is to the first... Initial visual embedding within a specific semantic subspace; superscript The subspace index used to define the subspace to which this feature belongs. This represents the mapping function of a multilayer perceptron network, which preserves the common underlying visual attributes that are prevalent in visual data during the projection process. This represents the visual block feature sequence matrix extracted and input into this layer in the preceding steps. This indicates that exclusive ownership corresponds to the first... A linear mapping layer function for semantic subspaces, where the subscripts Characterizing fully connected linear operations, superscript Characterize the function and the first The system establishes a one-to-one correspondence between subspaces; it is responsible for orienting the input common visual features to align them with the specific semantic dimension exclusive to the current subspace.

[0034] S3. Within each semantic subspace, construct semantic prototypes based on local descriptive primitives, perform multi-head cross-attention interaction between visual features and semantic prototypes, fuse global semantic anchors, and generate a multi-view visual embedding set. Furthermore, in S3, semantic prototypes are constructed based on local descriptive primitives, and multi-head cross-attention interactions are performed between visual features and semantic prototypes, including: The semantic prototype is obtained by using local descriptive primitives as query keys and values ​​for attention aggregation operations. Within each semantic subspace, the initial visual embedding is used as the query vector, and the semantic prototype is used as the key vector and value vector to perform multi-head cross-attention operation to obtain the visual features after interaction.

[0035] In S3, the fusion of global semantic anchors to generate a multi-view visual embedding set includes: Max pooling is performed on the visual features after interaction using a scaling factor, while average pooling is performed on the initial visual embedding. The features processed by max pooling and the features processed by average pooling are weighted by residual fusion to obtain aggregated visual features. The aggregated visual features and global semantic anchors are concatenated and normalized to generate the final visual embedding for a specific semantic subspace. The final visual embeddings of all specific semantic subspaces are combined into a multi-view visual embedding set.

[0036] Specifically, this step aims to address the technical problem of visual features lacking clear semantic interpretation. The previously extracted local textual descriptive primitives are aggregated into structured semantic prototypes. Within each specific semantic subspace, these semantic prototypes are used as key-value pairs to perform multi-head cross-attention interactions on the visual features. This process forces the visual features to focus on the semantic region most relevant to the current viewpoint. Subsequent residual pooling and complementary fusion with global semantic anchors effectively prevent semantic drift, ensuring that the final visual embeddings for each viewpoint possess both local discriminative power and alignment with the macroscopic semantic benchmark.

[0037] In the semantic prototype aggregation stage, a set of modality-independent learnable parameters is defined as latent semantic basis vectors. These learnable parameters are used as query vectors, and pre-extracted and linearly projected local description primitives are used as key and value vectors. Multi-head cross-attention aggregation and layer normalization are then performed to extract core semantic attributes and generate semantic prototypes. The calculation formula for this process is as follows: ; ; ; in This represents the local descriptive primitive matrix after projection. This represents the linear projection function performed on the local descriptive primitives. This represents the local descriptive primitive sequence features of the preceding input. It belongs to the symbol. Indicates size is A space of real-valued feature matrices of dimension 1. This represents the intermediate semantic features after cross-attention aggregation. This represents the multi-head cross-attention operation function. Let represent a set of modality-independent learnable parameters defined and used as latent semantic basis vectors, which are in ... A real space of dimensions. This indicates the core semantic prototype obtained through final refinement, which is in the following position: A real space of dimensions. The representation layer normalization processing function. This represents the linear projection function performed on intermediate semantic features.

[0038] Entering the multi-view visual semantic interaction stage. In order to generate the... The characteristics of a specific view, in the first Within a semantic subspace, the initial visual embedding of that subspace is used as the query vector, and the generated semantic prototypes are used as the key and value vectors. Cross-attention is used to activate the most relevant semantic region for the current viewpoint, followed by linear projection and normalization. The calculation formula is as follows: ; ; in Indicates the first Intermediate visual features of a specific view after cross-attention interaction. Indicates the first Initial visual embedding of semantic subspaces. This represents the semantic prototype of the input. Indicates the first The semantically rich visual features output by a specific view. This represents a linear projection function.

[0039] Residual pooling fusion is then performed. A residual pooling strategy is used to weight and fuse the semantically rich visual features with the original initial visual embedding, generating a compact vector and preventing semantic drift. The calculation formula is as follows: ; in This represents the compact visual features obtained after fusion, located in dimension . The real number space. This represents a learnable scaling factor. This represents the maximum pooling function. This represents the initial visual embedding of the input. This represents the average pooling function. It represents visual features rich in semantics.

[0040] Finally, global semantic anchor point complementarity fusion is performed. The previously acquired global semantic anchor points and compact visual features are introduced for feature concatenation, linear projection, and normalization to output the final view features. The calculation formula is as follows: ; in Indicates the first The final visual embedding generated from each view is located in dimension [ ]. The real number space. This represents the L2 normalization processing function. This represents the fused projection matrix, located at a size of The space of real matrix numbers. It indicates a compact visual feature. Symbols indicating feature splicing operations. This represents the global semantic anchor of the input.

[0041] S4. Aggregate the multi-view visual embedding set to generate visual centroids, push the visual centroids into the cross-view centroid pool, and calculate the global discrimination loss based on the centroid features and text modal features in the cross-view centroid pool. Furthermore, in S4, aggregating the multi-view visual embedding set to generate visual centroids and pushing the visual centroids into the cross-view centroid pool includes: The visual centroid is generated by summing and averaging the features of all views in the multi-view visual embedding set. An exponential moving average strategy is used to update the network parameters of the key visual encoders within the retrieval network. The visual centroid is stored in the cross-view centroid pool maintained by the key visual encoder.

[0042] In S4, the global discriminant loss is calculated based on centroid features and text modality features in the cross-view centroid pool, including: The positive sample similarity between centroid features in the cross-view centroid pool and the matched key text features is calculated based on the cosine similarity function. The negative sample similarity between centroid features in the cross-view centroid pool and negative sample features in the text sample pool is calculated based on the cosine similarity function. The global discriminant loss from image to text and the global discriminant loss from text to image are calculated based on the similarity of positive and negative samples. The global discriminant loss is obtained by adding the global discriminant loss from image to text to the global discriminant loss from text to image.

[0043] Specifically, this step is used to establish cross-modal alignment associations in the multi-view subspace and construct a negative sample feature queue. Directly storing all view features into the negative sample pool can cause memory bottlenecks and introduce view-specific noise. By adopting a centroid-based pooling approach, multi-view features are aggregated into a single visual centroid. Combined with independently updated key encoder parameters, this allows for an expansion of the number of negative samples across batches while controlling hardware resource consumption.

[0044] Entering the key encoder parameter update phase. To construct stable negative sample features, an exponential moving average strategy is invoked to update the parameters of the key visual encoder. Let the current training batch size be... , No. The parameter update formula for the key visual encoder in the next iteration is: ; in Indicates the first The key network parameters of the visual encoder during the next iteration. This indicates the set momentum coefficient. Indicates the first The key network parameters of the visual encoder during the next iteration. Indicates the first The network parameters of the master encoder during the next iteration.

[0045] The process then moves to the centroid aggregation and pooling stage. Before updating the sample pool with features, the embedding sets generated by the key visual encoders are extracted, and a summation and averaging operation is performed on the features of each view to generate the visual centroid. For the [specific feature]... The formula for calculating the visual centroid of a sample is: ; in This represents the visual centroid generated by the aggregation. Indicates the preset number of views. Indicates the first in the current training batch The first sample The key visual embeddings of each view.

[0046] After obtaining the visual centroid, the visual centroid is... The corresponding key text embeddings are pushed into the visual centroid pool. and text sample pool Among them and These represent the sizes of the visual centroid pool and the text sample pool, respectively.

[0047] The next step is to calculate the global discriminative loss. To endow each semantic subspace with independent discriminative power and utilize negative samples in the pooling, a cosine similarity function is used to measure the distance between features, and the global discriminative loss from image to text is calculated. The calculation formula is as follows: ; in This represents the calculated global discriminative loss from image to text. This indicates the training batch size. Indicates the first in the current training batch The first sample Visual embedding of a view. Indicates and The corresponding key text is embedded. This represents a negative sample in the text sample pool. This indicates the preset temperature coefficient. This represents the cosine similarity function.

[0048] Based on symmetric computation logic, the global discriminative loss from text to image is calculated. The calculation formula is as follows: ; in This represents the calculated global discriminative loss from text to image. This represents the text feature embedding in the current training batch. Indicates and The corresponding number The key visual embeddings of each view. This represents the negative sample of the visual centroid in the visual centroid pool.

[0049] Finally, the loss terms from both modalities are directly summed to obtain the final global discriminative loss. The calculation formula is: ; in This represents the final global discriminative loss, which forces the system to align features in an independent semantic subspace. The output error result is then passed to the network's parameter iteration module.

[0050] S5. For a multi-view visual embedding set, calculate the dense alignment loss and the maximum alignment loss to perform multi-view alignment; Furthermore, in S5, calculating the dense alignment loss includes: Extract the most difficult negative sample text from the training batch based on the global maximum similarity matching criterion; The dense alignment loss is calculated based on the similarity between the final visual embedding of a specific semantic subspace and the matching key text features, the similarity between the final visual embedding of a specific semantic subspace and the most difficult negative sample text, and the margin boundary value.

[0051] In S5, calculating the maximization of the alignment loss includes: Calculate the similarity between all view features and key text features in the multi-view visual embedding set; Filter out the highest similarity response value among all similarities; Construct a maximum alignment loss based on the maximum similarity response value.

[0052] Specifically, this step establishes a fine-grained association between visual multi-views and textual semantics through a dual loss constraint mechanism. Dense alignment loss, by mining the globally most difficult negative samples within the training batch, forces each subspace view to accurately distinguish positive and negative sample pairs in the feature space, thus addressing the gradient sparsity problem during backpropagation. Maximizing alignment loss, by mining view-specific difficult negative samples, implicitly encourages different views to focus on complementary local semantic details in the image, thereby enhancing the robustness of feature representation.

[0053] Before performing alignment calculations, the maximum matching similarity between images and text in a multi-view environment must first be defined. The system extracts all view features from the multi-view visual embedding set, calculates their cosine similarity with the corresponding text features, and selects the highest response value. The calculation formula for this process is as follows: ; in Indicates the first A picture and text pair in Text features in each view The highest response similarity value; This represents the function that takes the maximum value. This represents the view's index, with values ​​increasing from 1 to... ; Indicates the first In the nth sample Visual embedding vectors corresponding to each view; Represents the matching text feature embedding vector; This represents the cosine similarity metric function.

[0054] The dense alignment loss calculation phase begins. First, based on the global maximum similarity matching criterion, the most difficult negative sample text is extracted from the current training batch. Its defined formula is: ; in This indicates that the value corresponding to the first [number]th [unit] is mined based on the global maximum similarity. The most difficult negative sample text for each image sample; This function returns the index of its argument when the function value reaches its maximum. This represents the total number of samples in the current training batch; Indicates the index number of the non-matching sample within the batch; Indicates the first The image sample and the first The highest response similarity between non-matching texts.

[0055] Subsequently, combining the most difficult negative sample and the boundary spacing constant, the dense alignment loss is calculated. The calculation formula is as follows: ; in This represents the final output density alignment loss value; This represents a pre-defined boundary interval constant; A mathematical operator that takes the maximum value between the internal calculation result and 0; Indicates the first The first sample Visual embedding of individual views; This represents the text features corresponding to positive samples.

[0056] The system then proceeds to the stage of maximizing alignment loss calculation. First, it independently filters the hard-to-bear sample text for each view to enhance semantic diversity among views. The formula for defining the hard-to-bear sample text for the $k$-th view is: ; in Indicates the first Specific hard-to-bearing sample text embeddings from each viewpoint. Based on this, a maximum alignment loss is constructed, the formula of which is: ; in This represents the maximum alignment loss value of the final output. This loss term requires that the maximum response value of each view to a positive sample must be significantly higher than its similarity to a specific negative sample of that view. The calculated dense alignment loss and maximum alignment loss are then output to step S6 to construct the final overall training objective function.

[0057] S6. Construct an objective function by combining global discrimination loss, dense alignment loss and maximum alignment loss. Train the retrieval network based on the objective function to obtain the image-text retrieval model. Furthermore, in S6, a target function is constructed by jointly using the global discriminative loss, dense alignment loss, and maximize alignment loss. The retrieval network is then trained based on this target function, resulting in an image-text retrieval model including: Hyperparameter weights are assigned to the global discrimination loss, the dense alignment loss, and the alignment maximization loss, respectively. The objective function is obtained by linearly summing the global discrimination loss, dense alignment loss, and maximize alignment loss after assigning hyperparameter weights. The image-text retrieval model is obtained by updating the network parameters of the retrieval network based on the objective function using the backpropagation algorithm until the retrieval network reaches a convergent state.

[0058] Specifically, this step achieves closed-loop updates of the retrieval network parameters by integrating supervision signals of different granularities. The global discriminative loss constrains the consistency of macroscopic semantics, while the dense alignment loss and the maximized alignment loss force the model to capture complementary detailed features from a microscopic perspective. This multi-objective collaborative optimization mechanism ensures that the retrieval network can take into account both global topological structure and local semantic relationships when facing complex image and text data, ultimately obtaining a retrieval model with high generalization ability.

[0059] The data input sources for the objective function construction phase include the global discriminative loss, dense alignment loss, and maximizing alignment loss calculated in previous steps. First, hyperparameter weights are assigned to each loss term. Based on the weight allocation strategy defined in the documentation, hyperparameter weights are assigned to the dense alignment loss. And assign corresponding weights to maximize the alignment loss. .

[0060] Then, a linear summation operation is performed to obtain the final overall training objective function. The specific calculation formula is as follows: ; in This represents the output value of the final constructed overall training objective function. This represents the global discriminant loss value generated by the preceding steps. This represents the pre-set balancing hyperparameters. This represents the dense alignment loss value calculated in the preceding steps. This represents the weighting coefficients assigned to maximize the alignment loss. This represents the maximum alignment loss value generated by the preceding steps.

[0061] Obtain the overall training objective function Next, the system enters the network parameter iteration phase. The calculated loss value is used as the error signal, and the gradient is calculated based on the backpropagation algorithm. The gradient information is then transferred to the learnable parameters of each layer of the retrieval network, executing parameter updates. This training process is repeated on a cross-modal dataset until the loss function curve of the retrieval network reaches a preset convergence state or a preset maximum number of iterations. After training terminates, the current network parameter state is saved, obtaining the final image-text retrieval model. This model can then be used to process image or text query commands to be retrieved, outputting the corresponding cross-modal matching results.

[0062] Example 2: Experimental verification and result analysis.

[0063] 1. Experimental Environment and Parameter Configuration To verify the effectiveness of the VLMs-based multi-view semantic topology alignment method described in this invention, this embodiment is executed under specific hardware and software conditions. The hardware configuration includes: an Intel Core i9-13900K processor, dual NVIDIA GeForce RTX 4090 graphics processors, and 128GB of memory. The software environment is based on the Ubuntu 22.04 operating system, and the deep learning framework is PyTorch 2.1.

[0064] During network training, an initial learning rate is set. Batch size Set to 128. This applies to the balancing hyperparameters in the objective function. In this embodiment, the value was set to 0.4 based on experimental testing.

[0065] 2. Experimental Data Collection and Processing This embodiment selects three public datasets in the cross-modal retrieval field to perform performance evaluation: The Flickr30K dataset contains 31,783 images, each matched with 5 manually annotated descriptive text sentences.

[0066] The MSCOCO5k dataset contains 123,287 images. This embodiment uses the standard 5k test set partitioning scheme.

[0067] Dataset 3: A cross-modal test set in a specific scenario is selected to verify the model's generalization ability.

[0068] During the data processing stage, the images are uniformly scaled to 384 by 384 pixels, and the text is processed by word segmentation through a pre-trained text encoder.

[0069] 3. Numerical Analysis and Comparative Analysis The M-STAR method proposed in this invention is compared with existing technologies under different metrics. The evaluation metric used is recall. (in ) and total recall rate .

[0070] Numerical analysis of the Flickr30K dataset: Table 1: ; ; Experimental results show that, in image retrieval text tasks, the method of this invention performs well. The indicator reached 88.7%, an improvement of 2.4 percentage points compared to the latest algorithm APSE-IPIK in 2025. In the total indicator... The present invention achieves a score of 552.5, significantly outperforming all comparable schemes. This verifies that the multi-view semantic topological alignment mechanism can effectively capture the deep correlation between images and text.

[0071] Numerical analysis of the MSCOCO5k dataset: Table 2: ; On larger datasets, the method of this invention demonstrates its effectiveness in retrieving images from text. The algorithm achieved a success rate of 50.8%, making it the only one among the compared methods to break the 50% mark. The numerical results demonstrate the robustness of the algorithm in large-scale scenarios.

[0072] Numerical analysis of the MSCOCO1K dataset: Table 3: ; On the standard 1K subset, the method of the present invention The value reached 535.6. Compared with the algorithms DSRLN and VSSR of the same year, this invention achieved numerical leadership in all sub-categories of recall metrics, proving its universality on datasets of different sizes.

[0073] 4. Analysis of the reduction experiment To further verify the contribution of each technical feature in the claims to the final technical effect, this embodiment performed a reduction experiment. Performance changes were observed by removing core modules one by one. Table 4: ; Remove semantic topological priors: The sharp drop from 552.5 to 531.6 demonstrates the necessity of constructing dual-granularity semantic features using a large language model to alleviate the semantic sparsity problem.

[0074] Removing global instance discrimination: that is, not performing centroid construction and global alignment in step S4, results in a significant performance drop, verifying the role of the centroid pool in maintaining consistency of negative samples outside the batch.

[0075] Removing the alignment loss term: the performance is still lower than the full model, which proves the synergistic optimization effect of dense alignment and maximizing alignment loss on multi-view semantic decoupling in step S5.

[0076] 5. Conclusion on Technical Effects In summary, numerical analysis and comparative experiments demonstrate that the method described in this invention offers a significant performance improvement over existing technologies. By introducing dual-granularity semantic priors and constraining multi-view topological alignment loss, the model can accurately locate local semantics in images while maintaining global topological consistency. This effectively solves the matching bias problem caused by semantic entanglement and brief descriptions in cross-modal retrieval, exhibiting high practical value and technological advancement.

[0077] like Figure 4 As shown, this embodiment further tests the overall performance of each comparative method in terms of both performance improvement and inference speedup. Traditional methods such as VSRN or GPO, while improving inference speed to some extent, often sacrifice retrieval accuracy; while some high-precision methods are limited by their enormous computational overhead. In contrast, the M-STAR method described in this invention... Figure 4 The optimal position in the upper right corner not only achieved a significant performance gain of over 20 points but also achieved an inference speedup of nearly 8 times that of the baseline model. This demonstrates that the offline semantic prior extraction and multi-view centroid pooling mechanism proposed in this invention effectively avoids complex online cross-modal interaction calculations while significantly improving retrieval accuracy, achieving a superior balance between accuracy and efficiency.

[0078] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image-text retrieval method based on VLMs with multi-view semantic topological alignment, characterized in that, Includes the following steps: S1. Obtain image-text pairs from the cross-modal dataset; expand the original text of the image-text pairs using a multimodal large language model, extract global semantic anchors and local descriptive primitives, and construct dual-granularity semantic features; S2. Extract the visual features of the original image in the image-text pair, and project the visual features into multiple semantic subspaces; S3. Within each of the semantic subspaces, construct a semantic prototype based on the local descriptive primitives, perform multi-head cross-attention interaction between the visual features and the semantic prototypes, fuse the global semantic anchors, and generate a multi-view visual embedding set. S4. Aggregate the multi-view visual embedding set to generate visual centroids, push the visual centroids into the cross-view centroid pool, and calculate the global discrimination loss based on the centroid features and text modal features in the cross-view centroid pool; S5. For the multi-view visual embedding set, calculate the dense alignment loss and the maximum alignment loss to perform multi-view alignment; S6. Construct an objective function by combining the global discrimination loss, the dense alignment loss, and the maximize alignment loss. Train the retrieval network based on the objective function to obtain an image-text retrieval model.

2. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S1, the step of expanding the original text of the image-text pair using a multimodal large language model, extracting global semantic anchors and local descriptive primitives, and constructing dual-granularity semantic features includes: The original text is expanded and described using a multimodal large language model with frozen network parameters. The global high-dimensional features of the extended description text are extracted by a text encoder and used as the global semantic anchor. Fine-grained sequence features of the extended description text are extracted as the local description primitives; The global semantic anchor and the local descriptive primitive are combined to form the dual-granularity semantic feature.

3. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S2, extracting the visual features of the original image in the image-text pair and projecting the visual features into multiple semantic subspaces includes: The original image is processed by feature encoding through a visual backbone network to obtain discrete visual block feature sequences. A multilayer perceptron network and a linear mapping layer corresponding to a specific semantic subspace are used to perform nonlinear projection processing on the visual block feature sequence to obtain an initial visual embedding specific to each semantic subspace.

4. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 3, characterized in that, In S3, the step of constructing a semantic prototype based on the local descriptive primitives and performing multi-head cross-attention interaction between the visual features and the semantic prototype includes: The semantic prototype is obtained by performing attention aggregation operation on the local description primitives as query keys and values. Within each of the semantic subspaces, the initial visual embedding is used as the query vector, and the semantic prototype is used as the key vector and value vector to perform multi-head cross-attention operation to obtain the visual features after interaction.

5. The image-text retrieval method based on VLMs with multi-view semantic topology alignment according to claim 4, characterized in that, In S3, fusing the global semantic anchors to generate a multi-view visual embedding set includes: Max pooling is performed on the visual features after interaction using a scaling factor, while average pooling is performed on the initial visual embedding. The features processed by max pooling and the features processed by average pooling are weighted by residual fusion to obtain aggregated visual features. The aggregated visual features and the global semantic anchors are combined and normalized to generate the final visual embedding of a specific semantic subspace. The final visual embeddings of all the specific semantic subspaces are combined into the multi-view visual embedding set.

6. The image-text retrieval method based on VLMs with multi-view semantic topology alignment according to claim 1, characterized in that, In S4, aggregating the multi-view visual embedding set to generate a visual centroid, and pushing the visual centroid into the cross-view centroid pool includes: The visual centroid is generated by summing and averaging all view features in the multi-view visual embedding set. An exponential moving average strategy is used to update the network parameters of the key visual encoders within the retrieval network. The visual centroid is stored in the cross-view centroid pool maintained by the key visual encoder.

7. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S4, the calculation of the global discrimination loss based on the centroid features and text modality features in the cross-view centroid pool includes: The positive sample similarity between the centroid features in the cross-view centroid pool and the matched key text features is calculated based on the cosine similarity function. The negative sample similarity between the centroid features in the cross-view centroid pool and the negative sample features in the text sample pool is calculated based on the cosine similarity function. The global discriminant loss from image to text and the global discriminant loss from text to image are calculated based on the positive sample similarity and the negative sample similarity. The global discrimination loss is obtained by adding the global discrimination loss from image to text to the global discrimination loss from text to image.

8. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S5, the calculation of dense alignment loss includes: Extract the most difficult negative sample text from the training batch based on the global maximum similarity matching criterion; The dense alignment loss is calculated based on the similarity between the final visual embedding of the specific semantic subspace and the matching key text features, the similarity between the final visual embedding of the specific semantic subspace and the most difficult negative sample text, and the interval boundary value.

9. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S5, the calculation of maximizing the alignment loss includes: Calculate the similarity between all view features and key text features in the multi-view visual embedding set; Filter out the maximum similarity response value among all the stated similarities; The maximum alignment loss is constructed based on the maximum similarity response value.

10. The image-text retrieval method based on VLMs with multi-view semantic topological alignment according to claim 1, characterized in that, In S6, a target function is constructed by combining the global discrimination loss, the dense alignment loss, and the maximized alignment loss. The retrieval network is trained based on the target function to obtain the image-text retrieval model, including: Hyperparameter weights are assigned to the global discrimination loss, the dense alignment loss, and the maximized alignment loss, respectively. The objective function is obtained by linearly summing the global discrimination loss, the dense alignment loss, and the maximized alignment loss after assigning the hyperparameter weights. Based on the objective function, the backpropagation algorithm is used to update the network parameters of the retrieval network until the retrieval network reaches a convergent state to obtain the image-text retrieval model.

Images