Multilevel collaborative selection method for matching images and texts based on linguistic guidance and computer system

Abstract

The application relates to the technical field of computer vision, in particular to a multilevel collaborative selection graph-text matching method based on linguistics guidance and a computer system. In one aspect, an interpretable word importance weight based on linguistics guidance is used as a control signal for guiding the whole process, cross-modal enhancement, adaptive region selection and feature dimension selection are realized, and the adaptability of a model to different text complexities is improved. In another aspect, a multilevel collaborative selection mechanism combined with region dimension and feature dimension is used to simultaneously inhibit irrelevant regions and redundant feature dimensions, and the fine-grained discrimination ability is improved. The application aims to solve the problem of how to reduce the misjudgment risk in the graph-text matching process of different text complexities.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a linguistically guided multi-level collaborative selection image-text matching method and computer system. Background Technology

[0002] Image-text cross-modal retrieval, also known as image-text matching, aims to map images and natural language descriptions to a comparable common representation space, thereby enabling functions such as text retrieval of images and image retrieval of text.

[0003] Traditional image-text matching methods often employ a dual-encoding structure: the image side extracts a global vector representation, while the text side encodes the sentence to obtain a global vector representation. A matching score is then calculated using cosine similarity, and training is performed using interval-based contrastive or ranking loss. This type of method offers high retrieval efficiency and facilitates offline database construction and rapid retrieval.

[0004] However, because this method compresses images and sentences into single vectors, fine-grained semantics are easily weakened during the compression process. In actual image-text matching with different text complexities, it is easy to encounter situations where semantically similar texts are not matched in detail but are still judged as highly similar.

[0005] In view of this, this application proposes a linguistically guided multi-level collaborative selection image-text matching method. On the one hand, it guides cross-modal processing of the text side through linguistics, thereby improving the compatibility of image-text matching with different text complexities. On the other hand, it suppresses irrelevant regions and redundant feature dimensions through an innovatively designed multi-level collaborative selection mechanism, thereby improving the model's fine-grained discrimination ability. Summary of the Invention

[0006] The main purpose of this application is to provide a linguistically guided multi-level collaborative selection image-text matching method, which aims to solve the problem of how to reduce the risk of misjudgment in the image-text matching process with different text complexities.

[0007] To achieve the above objectives, this application provides a linguistically guided multi-level collaborative selection image-text matching method, the method comprising:

[0008] S10, preprocess and encode the input image and input text respectively to obtain the image region feature vector set and the text word-level feature sequence;

[0009] S20, generate word importance weight values ​​for each word in the input text based on the text word-level feature sequence, wherein the word importance weight values ​​are obtained by weighting a preset part-of-speech prior weight, inverse document frequency weight, and multilayer perceptron network weights generated based on the text word-level feature sequence;

[0010] S30, the text word-level feature sequence is weighted and aggregated according to the word importance weight value and mapped to the visual space to generate text global features, and the image global features corresponding to the image region feature vector set are calculated. The text global features are concatenated with the image global features and the image region feature vector set respectively, and the joint importance tensor is obtained after learning through a gating network.

[0011] S40, based on the joint importance tensor and the word importance weight value, filter the enhanced image features in the image region feature vector set;

[0012] S50, the text word-level feature sequence is segmented, fused, and updated according to the enhanced image features to obtain the enhanced text word-level feature sequence;

[0013] S60, construct an image-text similarity matrix based on the enhanced text word-level feature sequence and the image region feature vector set, and perform image-text matching based on the image-text similarity matrix.

[0014] Optionally, in step S10, the preprocessing encoding includes:

[0015] S11, the input text is segmented and indexed to obtain the word identifier sequence and corresponding length information of the text, and the word identifier sequence is padded to align to a preset maximum length;

[0016] S12, embed the word identifier sequence after padding into word vectors, and perform context modeling on the embedded sequence based on a context encoding network to obtain the text word-level feature sequence;

[0017] And, S13, determine the candidate salient regions in the input image, and extract visual features from each salient region and map them to a preset feature dimension to obtain a preset number of image region feature vectors.

[0018] Optionally, S20 specifically includes:

[0019] S21, based on the word identifiers in the text word-level feature sequence, extract word vector features, part-of-speech embedding features, absolute position embedding features and relative position embedding features respectively, and concatenate the four types of features to obtain a multi-source linguistic feature sequence;

[0020] Specifically, the part-of-speech embedding feature is obtained by acquiring the part-of-speech tags corresponding to each word through a pre-constructed part-of-speech dictionary and embedding them into a fixed dimension; the absolute position embedding feature is obtained by position encoding based on the absolute position index of the word in the sentence; and the relative position embedding feature is obtained by encoding based on the offset of the word relative to the center position of the sentence.

[0021] S22, the multi-source linguistic feature sequence is input into a bidirectional gated recurrent unit network for bidirectional context encoding to obtain a context-aware linguistic feature representation;

[0022] S23, apply a multi-head self-attention mechanism to model long-distance dependencies on the context-aware linguistic feature representation, and perform residual connections with the original features to obtain an enhanced linguistic feature representation. ;

[0023] S24, the enhanced linguistic feature representation is input into a multilayer perceptron network, and the multilayer perceptron network weights of each word are obtained. :

[0024]

[0025] In the formula: Represents a multilayer perceptron network;

[0026] S25, obtain the first Part-of-speech prior weights of each word :

[0027] Among them, the part-of-speech prior weight For the first The preset importance weight associated with the part-of-speech category of each word in the pre-constructed part-of-speech dictionary;

[0028] And, calculate the first Inverse document frequency weight of each word :

[0029]

[0030] In the formula: The total number of documents in the training corpus; For including the first The number of documents per word;

[0031] S26, based on the multilayer perceptron network weights, part-of-speech prior weights, and inverse document frequency weights... Calculate the first Word importance weight value :

[0032]

[0033] in: , , The learnable fusion coefficient;

[0034] The fusion coefficients are adaptively updated through backpropagation during training, and normalization constraints are applied before fusion to ensure that all fusion coefficients are non-negative and their sum is 1.

[0035] Optionally, the expression for the joint importance tensor is:

[0036]

[0037] In the formula, Represents the joint importance tensor; This means that the features of each image region are concatenated with the global features of the text and then input into the spatial gating network, and the output is the first... The spatial importance of each region; This means that the global features of the image and the global features of the text are concatenated and then input into the channel-gated network, and the output is the first... The importance of each feature in the channel;

[0038] in:

[0039]

[0040]

[0041] In the formula, Represents global features of the text; Represents global features of an image; For the first Feature vectors of each image region; Indicates feature splicing; For channel gating network; It is the Sigmoid activation function. For spatial gating networks.

[0042] Optionally, the text global features The calculation expression is:

[0043]

[0044] In the formula, For the first Textual features of each word; Sentence length; For linear projection layers; Indicates the first Each word's importance weight;

[0045] The global features of the image The calculation expression is:

[0046]

[0047] In the formula, This represents the total number of image regions.

[0048] S40 includes:

[0049] S41, normalize the word importance weights into a probability distribution. And calculate its information entropy as semantic complexity. Among them, the larger the information entropy value, the more complex the semantics of the text:

[0050]

[0051]

[0052] S42, normalize the semantic complexity to obtain and according to Determine the number of regions to retain. Number of features retained :

[0053]

[0054]

[0055]

[0056] In the formula: and These are the minimum and maximum entropy values ​​within the current batch, respectively. and These represent the maximum and minimum number of regions that can be retained in the region dimension, respectively. and These represent the maximum and minimum number of feature dimensions that can be retained, respectively. Indicates rounding down;

[0057] S43, averaging the joint importance tensor along the feature dimension to obtain the first... The overall importance weight of each region Select the one with the highest weight One region is assigned a weight of 1.0, and the remaining regions are assigned a decay weight. :

[0058]

[0059]

[0060] In the formula: For area soft mask; The attenuation coefficient; For feature dimensions;

[0061] S44, averaging the joint importance tensor along the region dimension to obtain the first... The combined importance weight of each feature dimension Select the one with the highest weight One feature dimension is assigned a weight of 1.0, and the remaining dimensions are assigned decaying weights. :

[0062]

[0063]

[0064] In the formula: For feature dimension soft mask;

[0065] S45, apply a soft mask to the region. With feature dimension soft mask Combined application to image region features yields enhanced image features after collaborative filtering. :

[0066]

[0067] In the formula: The first feature in the original image region The first region The values ​​of each feature dimension.

[0068] Optionally, S50 includes;

[0069] S51, Calculate the self-similarity matrix between image regions based on enhanced image features. and text-image cross-modal similarity matrix :

[0070]

[0071] In the formula, This is the feature matrix of the original image region; This is the enhanced image feature matrix after collaborative filtering; This is the text word-level feature matrix;

[0072] S52, perform learnable projection transformations on the self-similarity matrix and the cross-modal similarity matrix respectively, and perform matrix multiplication to obtain the cross-modal structural propagation features. :

[0073]

[0074] In the formula: For learnable projection transformation;

[0075] S53, calculate and standardize the statistics of the cross-modal structural propagation features, and classify the word-level features into three categories: stable region, boundary region, and noise region based on the absolute value of the standardization result:

[0076]

[0077] in, For the stable region, For the boundary area, This is a noisy area;

[0078] In the formula, The mean of the cross-modal structural propagation characteristics; The standard deviation of the cross-modal structure propagation characteristics;

[0079] S54, the cross-modal structural propagation features and the text word-level feature sequence are weighted and fused using different fusion coefficients for the three types of regions to obtain the enhanced text word-level feature sequence. :

[0080]

[0081] In the formula, It is a text word-level feature sequence; , , The fusion coefficient for the three types of regions; , This is the threshold for region division.

[0082] Optionally, S60 includes:

[0083] S61, calculate the... The text and the first Word-region attention matrix for each image :

[0084]

[0085] In the formula: No. Enhanced text word-level feature sequences of a text; For the first Features of an image;

[0086] S62, averaging the attention matrix by word dimension to obtain the [number]th [word]. The text and the first Similarity of images :

[0087]

[0088] In the formula: Attention matrix The Row vector, representing the first row vector. The first text The word for the first Attention weight vectors for all regions of an image; For the first The length of each text; This is an attention aggregation function that includes weighted positive and negative similarity.

[0089] S63, sort the candidate images or candidate texts according to the similarity and output the image-text matching results.

[0090] Optionally, after S60, the following is also included:

[0091] The image-text matching results are optimized using a bidirectional ranking loss function, the expression of which is:

[0092]

[0093] in:

[0094]

[0095]

[0096] In the formula, For image-to-text retrieval loss; For text-to-image retrieval loss; These are boundary hyperparameters; Indicates the first The similarity between an image and its corresponding matching text. Indicates the first The similarity between a text and its corresponding matching image.

[0097] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the linguistically guided multi-level collaborative selection graph-text matching method as described in any of the preceding claims.

[0098] This application has at least the following beneficial effects:

[0099] 1. Utilize interpretable word importance weights (integrating part-of-speech, inverse document frequency, and positional information) as control signals to guide the entire process, achieving cross-modal enhancement, adaptive region selection, and feature dimension selection, thereby improving the model's adaptability to different text complexities;

[0100] 2. By using a joint collaborative selection mechanism of region dimension and feature dimension, irrelevant regions and redundant feature dimensions are suppressed simultaneously, significantly reducing noise interference and improving fine-grained discrimination capability;

[0101] 3. Through simulation experiments, the total recall rSum index reached 536.5 and 544.9 on the Flickr30K and MS-COCO datasets, respectively. Attached Figure Description

[0102] Figure 1 This is a flowchart illustrating the linguistically guided multi-level collaborative selection image-text matching method involved in the embodiments of this application;

[0103] Figure 2 This is a schematic diagram illustrating the semantic importance weight distribution of words involved in the embodiments of this application;

[0104] Figure 3 This is a schematic diagram of the joint importance tensor of region and feature dimensions and soft mask filtering involved in the embodiments of this application;

[0105] Figure 4 This is a comparison chart of training loss involved in the embodiments of this application;

[0106] Figure 5 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0107] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0108] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0109] First Embodiment

[0110] Reference Figure 1 This embodiment provides a linguistically guided multi-level collaborative selection image-text matching method. In this embodiment, the execution subject of the method is an image-text matching model (hereinafter referred to as the model) capable of implementing the following steps. The method includes the following steps:

[0111] S10, preprocess and encode the input image and input text respectively to obtain the image region feature vector set and the text word-level feature sequence;

[0112] In this embodiment, the input image is processed into a set of image region feature vectors consisting of multiple image region feature vectors, and the input text is processed into a sequence of text word-level feature vectors consisting of multiple text word-level feature vectors.

[0113] Specifically, preprocessing encoding includes:

[0114] S11, the input text is segmented and indexed to obtain the word identifier sequence and corresponding length information of the text, and the word identifier sequence is padded to align to a preset maximum length;

[0115] In some optional implementations, in step S11, the input natural language description text is first segmented into word sequences; then, each word is converted into a corresponding integer index based on a pre-built vocabulary to form a word identifier sequence; at the same time, the actual effective length of the sequence is recorded; finally, zero-padding is performed on sequences whose length is less than the preset maximum length to facilitate batch calculation.

[0116] S12, embed the word identifier sequence after padding into word vectors, and perform context modeling on the embedded sequence based on a context encoding network to obtain the text word-level feature sequence;

[0117] In some alternative implementations, the word identifier sequence is vectorized using pre-trained GloVe word embeddings to obtain a word embedding sequence; then the word embedding sequence is input into a bidirectional gated recurrent unit network for context modeling to obtain a text word-level feature sequence.

[0118] And, S13, determine the candidate salient regions in the input image, and extract visual features from each salient region and map them to a preset feature dimension to obtain a preset number of image region feature vectors.

[0119] In some alternative implementations, a pre-trained Faster R-CNN model is used to detect the input image and obtain candidate salient regions; visual features are extracted from each salient region and mapped to a preset feature dimension to obtain a preset number of image region feature vectors.

[0120] Preferably, the preset number is 36 regions, and the feature dimension is 1024.

[0121] S20, generate word importance weight values ​​for each word in the input text based on the text word-level feature sequence, wherein the word importance weight values ​​are obtained by weighting a preset part-of-speech prior weight, inverse document frequency weight, and multilayer perceptron network weights generated based on the text word-level feature sequence;

[0122] In step S20, we introduce multi-source linguistic knowledge to guide the model to quantify the semantic importance weight of each word in the text, i.e., the word importance weight value, as a control signal for subsequent cross-modal processing.

[0123] Specifically, the generation of word importance weight values ​​is achieved through the following steps:

[0124] S21, based on the word identifiers in the text word-level feature sequence, extract word vector features, part-of-speech embedding features, absolute position embedding features and relative position embedding features respectively, and concatenate the four types of features to obtain a multi-source linguistic feature sequence;

[0125] Specifically, the part-of-speech embedding feature is obtained by acquiring the part-of-speech tags corresponding to each word through a pre-constructed part-of-speech dictionary and embedding them into a fixed dimension; the absolute position embedding feature is obtained by position encoding based on the absolute position index of the word in the sentence; and the relative position embedding feature is obtained by encoding based on the offset of the word relative to the center position of the sentence.

[0126] In some alternative implementations, the four types of features are extracted in the following ways:

[0127] (1) Word vector features: derived from pre-trained GloVe embeddings, with a dimension of 300;

[0128] (2) Part-of-speech embedding features: obtain the part-of-speech tags corresponding to each word through a pre-constructed part-of-speech dictionary, divide the part of speech into 8 categories (noun, verb, adjective, adverb, preposition, conjunction, pronoun, others), and embed them into a 64-dimensional vector;

[0129] (3) Absolute position embedding features: Position encoding is performed based on the absolute position index of the word in the sentence, and the embedding is embedded into a 32-dimensional vector;

[0130] (4) Relative position embedding features: The offset of a word relative to the center of the sentence is encoded, and the offset is mapped to a preset range and then embedded into a 32-dimensional vector.

[0131] S22, the multi-source linguistic feature sequence is input into a bidirectional gated recurrent unit network for bidirectional context encoding to obtain a context-aware linguistic feature representation;

[0132] In some alternative implementations, a two-layer bidirectional GRU network with a hidden layer dimension of 256 and a dropout rate of 0.3 is used to bidirectionally encode multi-source linguistic feature sequences, outputting a context-aware feature representation with a dimension of 512. .

[0133] S23, apply a multi-head self-attention mechanism to model long-distance dependencies on the context-aware linguistic feature representation, and perform residual connections with the original features to obtain an enhanced linguistic feature representation. ;

[0134] In some alternative implementations, an 8-head self-attention mechanism is used with an embedding dimension of 512 and a dropout rate of 0.2 to perform self-attention calculation on context-aware features; the self-attention output is added to the original context-aware features by residual addition to obtain an enhanced linguistic feature representation.

[0135] S24, the enhanced linguistic feature representation is input into a multilayer perceptron network, and the multilayer perceptron network weights of each word are obtained. :

[0136]

[0137] In the formula: Represents a multilayer perceptron network;

[0138] In some alternative implementations, the multilayer perceptron network (MLP) contains three fully connected layers that map 512-dimensional features to 256-dimensional, 128-dimensional, and 1-dimensional features respectively, with layer normalization, ReLU activation function, and dropout regularization used between the layers.

[0139] S25, obtain the first Part-of-speech prior weights of each word :

[0140] Among them, the part-of-speech prior weight For the first The preset importance weight associated with the part-of-speech category of each word in the pre-constructed part-of-speech dictionary;

[0141] And, calculate the first Inverse document frequency weight of each word :

[0142]

[0143] In the formula: The total number of documents in the training corpus; For including the first The number of documents per word;

[0144] Optionally, the preset importance weights corresponding to different parts of speech categories in step S25 are set based on linguistic rules, with higher weights assigned to parts of speech with stronger semantic carrying capacity and lower weights assigned to parts of speech with stronger semantic functionality.

[0145] S26, based on the multilayer perceptron network weights, part-of-speech prior weights, and inverse document frequency weights... Calculate the first Word importance weight value :

[0146]

[0147] in: , , These are learnable fusion coefficients used to adjust the relative contributions of the original importance weights, part-of-speech prior weights, and inverse document frequency weights during the fusion process.

[0148] The fusion coefficients are adaptively updated via backpropagation during training and are normalized before fusion to ensure that all fusion coefficients are non-negative and their sum is 1. This is done to guarantee that importance information from different sources can be weighted and fused in a stable and interpretable manner.

[0149] For example, see Figure 2 The diagram illustrates the distribution of word importance weights extracted from the text word-level feature sequence in a specific embodiment. For the input text:

[0150] A word-level semantic importance analysis was performed on the sentence "A man in a blue shirt standing in a garden". The horizontal axis represents the word sequence of the input text, and the vertical axis represents the semantic importance weight of each word. Different fill patterns represent different parts of speech categories.

[0151] As shown in the figure, core semantic words such as man, garden, shirt, standing, and blue have higher importance weights, while function words such as the article "a" and the preposition "in" have lower weights. This weight distribution conforms to linguistic principles, namely that content words such as nouns and verbs carry the main semantic information, while function words contribute less semantic information.

[0152] S30, the text word-level feature sequence is weighted and aggregated according to the word importance weight value and mapped to the visual space to generate text global features, and the image global features corresponding to the image region feature vector set are calculated. The text global features are concatenated with the image global features and the image region feature vector set respectively, and the joint importance tensor is obtained after learning through a gating network.

[0153] In step S30, a joint importance representation of region and feature dimensions is constructed under the guidance of the word importance weight values ​​obtained in the previous step. In this step, channel dimension importance and spatial region importance are learned separately through a decoupled two-stream gating mechanism, and a two-dimensional joint importance tensor is constructed through outer product operation to achieve joint modeling of the importance of region and feature at both levels, which is the multi-level collaborative selection described in this application.

[0154] Specifically, step S30 includes:

[0155] S31, the text word-level feature sequence is weighted and aggregated based on the importance of the text words to obtain the text global features. And it is mapped to the visual space through a linear projection layer:

[0156]

[0157] In the formula: For the first Textual features of each word; Sentence length; It is a linear projection layer.

[0158] This step uses word importance-weighted aggregation to make the global text features reflect more of the semantic core words, rather than simple average pooling.

[0159] Calculate global features of an image :

[0160]

[0161] In the formula: For the first Feature vectors of each image region; This represents the total number of image regions.

[0162] S32, the importance of feature dimensions is learned through a channel-gated network. Specifically, the global features of the image and the global features of the text are concatenated and then input into the channel-gated network, which outputs the first... Channel importance of each feature :

[0163]

[0164] In the formula: Indicates feature splicing; For channel gating network; This is the Sigmoid activation function.

[0165] Specifically, the channel-gated network consists of two fully connected layers. This design learns which feature dimensions (such as color, texture, shape, etc.) are more discriminative under the current text semantics through the interaction of global graph and text information.

[0166] S33, learning region importance through a spatial gating network, specifically: concatenating the features of each image region with the global text features and inputting the concatenation into the spatial gating network, outputting the first... Spatial importance of each region :

[0167]

[0168] In the formula: For spatial gating networks.

[0169] Specifically, the space gating network consists of two fully connected layers.

[0170] This design learns which image regions are more relevant to the semantics of the current text by interacting with local regions and the global text.

[0171] S34 performs an outer product operation on spatial importance and channel importance to obtain a two-dimensional joint importance tensor of region and feature dimensions. :

[0172]

[0173] For example, such as Figure 3 The diagram illustrates the construction process of the joint importance tensor of regions and feature dimensions. The left vertical bar chart represents the spatial importance vector of each image region, where regions highly relevant to text semantics have higher importance values, while background regions unrelated to text have lower importance values. The top horizontal bar chart represents the channel importance vector of each feature dimension, where feature dimensions encoding semantic information have higher importance values, while dimensions encoding low-level visual features have lower importance values.

[0174] S40, based on the joint importance tensor and the word importance weight value, filter the enhanced image features in the image region feature vector set;

[0175] In this step, the number of elements to be retained at each level is adaptively determined based on the semantic complexity of the text, and a soft masking mechanism is used to retain high-importance positions and suppress low-importance positions.

[0176] Specifically, the following steps are included:

[0177] S41, normalize the word importance weights into a probability distribution. And calculate its information entropy as semantic complexity. Among them, the larger the information entropy value, the more complex the semantics of the text:

[0178]

[0179]

[0180] In step S41, this design indicates that when the word importance distribution is relatively uniform (high entropy value), it means that the text contains multiple semantic focuses and the semantics are relatively complex; when the word importance is concentrated in a few words (low entropy value), it means that the text semantics are relatively simple and clear.

[0181] S42, normalize the semantic complexity to obtain and according to Determine the number of regions to retain. Number of features retained :

[0182]

[0183]

[0184]

[0185] In the formula: and These are the minimum and maximum entropy values ​​within the current batch, respectively. and These represent the maximum and minimum number of regions that can be retained in the region dimension, respectively. and These represent the maximum and minimum number of feature dimensions that can be retained, respectively. Indicates rounding down;

[0186] In some alternative implementations, Set to 10, Set to 30; set to , Set as , This design uses a joint embedding dimension. It allows semantically complex text to retain more regions and feature dimensions to capture multifocal information, while semantically simple text is filtered to suppress noise interference.

[0187] S43, averaging the joint importance tensor along the feature dimension to obtain the first... The overall importance weight of each region Select the one with the highest weight One region is assigned a weight of 1.0, and the remaining regions are assigned a decay weight. :

[0188]

[0189]

[0190] In the formula: For area soft mask; The attenuation coefficient; For feature dimensions;

[0191] S44, averaging the joint importance tensor along the region dimension to obtain the first... The combined importance weight of each feature dimension Select the one with the highest weight One feature dimension is assigned a weight of 1.0, and the remaining dimensions are assigned decaying weights. :

[0192]

[0193]

[0194] In the formula: The feature dimension is a soft mask; using a soft mask can preserve weak information flow, avoid gradient truncation, and improve training stability.

[0195] S45, apply a soft mask to the region. With feature dimension soft mask Combined application to image region features yields enhanced image features after collaborative filtering. :

[0196]

[0197] In the formula, The first feature in the original image region The first region The values ​​of each feature dimension.

[0198] This dual soft mask collaborative screening mechanism adaptively suppresses both the region and feature dimensions, effectively reducing the interference of irrelevant regions and redundant feature dimensions on subsequent matching.

[0199] S50, the text word-level feature sequence is segmented, fused, and updated according to the enhanced image features to obtain the enhanced text word-level feature sequence;

[0200] In step S50, a cross-modal enhancement term is constructed based on the region features after soft masking and collaborative weighting. The text word-level features are then segmented and fused according to this enhancement term to obtain the enhanced text word-level feature sequence. In this step, the enhancement term is constructed through a cross-modal structure propagation mechanism, and the text features are stably updated based on a segmented fusion strategy.

[0201] Further and optionally, it specifically includes the following steps:

[0202] S51, Calculate the self-similarity matrix between image regions based on enhanced image features. and text-image cross-modal similarity matrix :

[0203]

[0204] In the formula, This is the feature matrix of the original image region; This is the enhanced image feature matrix after collaborative filtering; This is the text word-level feature matrix;

[0205] S52, perform learnable projection transformations on the self-similarity matrix and the cross-modal similarity matrix respectively, and perform matrix multiplication to obtain the cross-modal structural propagation features. :

[0206]

[0207] In the formula: For learnable projection transformation;

[0208] S53, calculate and standardize the statistics of the cross-modal structural propagation features, and classify the word-level features into three categories: stable region, boundary region, and noise region based on the absolute value of the standardization result:

[0209]

[0210] in, For the stable region, For the boundary area, This is a noisy area;

[0211] In the formula, The mean of the cross-modal structural propagation characteristics; The standard deviation of the cross-modal structure propagation characteristics;

[0212] S54, the cross-modal structural propagation features and the text word-level feature sequence are weighted and fused using different fusion coefficients for the three types of regions to obtain the enhanced text word-level feature sequence. :

[0213]

[0214] In the formula: , , The fusion coefficient for the three types of regions; , This is the threshold for region division.

[0215] It should be noted that the design purpose of this segmented fusion strategy is as follows: for words in stable regions, the enhancement terms are strongly fused with the original features to introduce cross-modal information; for words in boundary regions, a conservative fusion strategy is adopted to avoid introducing noise; for words in other regions, the original features are preserved unchanged.

[0216] S60, construct an image-text similarity matrix based on the enhanced text word-level feature sequence and the image region feature vector set, and perform image-text matching based on the image-text similarity matrix.

[0217] In this embodiment, S60 includes:

[0218] S61, calculate the... The text and the first Word-region attention matrix for each image :

[0219]

[0220] In the formula: No. Enhanced text word-level feature sequences of a text; For the first Feature vectors of image regions;

[0221] S62, averaging the attention matrix by word dimension to obtain the [number]th [word]. The text and the first Similarity of images :

[0222]

[0223] In the formula: Attention matrix The Row vector, representing the first row vector. The first text The word for the first Attention weight vectors for all regions of an image; For the first The length of each text; This is an attention aggregation function that includes weighted positive and negative similarity.

[0224] S63, sort the candidate images or candidate texts according to the similarity and output the image-text matching results.

[0225] Furthermore, and optionally, to ensure the accuracy of the output image-text matching results, a bidirectional ranking loss function is used to optimize the image-text matching results. The expression of the bidirectional ranking loss function is as follows:

[0226]

[0227] in:

[0228]

[0229]

[0230] In the formula, For image-to-text retrieval loss; For text-to-image retrieval loss; These are boundary hyperparameters; Indicates the first The similarity between an image and its corresponding matching text. Indicates the first The similarity between a text and its corresponding matching image.

[0231] In the technical solution provided in this embodiment, on the one hand, interpretable word importance weights based on linguistic guidance are used as control signals to guide the entire process, thereby achieving cross-modal enhancement, adaptive region selection, and feature dimension selection, thus improving the model's adaptability to different text complexities; on the other hand, through a multi-level collaborative selection mechanism that combines region dimensions and feature dimensions, irrelevant regions and redundant feature dimensions are suppressed simultaneously, thereby improving fine-grained discrimination capability.

[0232] Verification Implementation Examples

[0233] Based on the first embodiment, this embodiment verifies the effectiveness of the linguistically guided multi-level collaborative selection image-text matching method. This embodiment evaluates the proposed method on two widely used public datasets:

[0234] Flickr30K: This dataset contains 31,000 images. Using standard data splitting methods, 29,000 images are used for training, 1,000 for validation, and 1,000 for testing. MSCOCO: This is a larger dataset with 123,287 images. It is split into 113,287 training images, 5,000 validation images, and 5,000 test images. For MSCOCO, we report the average performance on the 1K test subset with 5 cross-validations.

[0235] The evaluation metric used is recall@K (where, The metrics include rSum, where rSum is the sum of six Recall metrics:

[0236]

[0237] In the experimental setup, our model (Our) employed Faster R-CNN (based on a ResNet101 backbone network) and was pre-trained on the Visual Genome dataset to extract image region features. Specifically, we selected the model with the highest confidence level. There are 10 regions, and the feature dimension of each region is 2,048.

[0238] GloVe was used as the text feature extractor to extract features. These features were then mapped to a uniform dimension through a fully connected network. , The value was set to 1,024. The model was optimized using the Adam optimizer with a learning rate of 0.0005. The batch size for Flickr30K was 128, and for MSCOCO it was 256. Both datasets were trained for 20 epochs, with scaling parameters set to [specific parameters to be filled in]. Set it to 20.

[0239] See Table 1 below for a comparison of the method of this invention with existing methods on the Flickr30K dataset:

[0240] Table 1. Comparison results of the method of the present invention and existing methods on the Flickr30K dataset.

[0241]

[0242] Additionally, see Table 2 below for a comparison of the method of this invention with existing methods on the MS-COCO dataset:

[0243] Table 2. Comparison results of the method of the present invention and existing methods on the MS-COCO dataset.

[0244]

[0245] The experimental results show that the method of this invention achieves significantly better performance than other models on both datasets: on the Flickr30K dataset, the rSum index reaches 536.5, which is 5.6 points higher than the suboptimal method MSCA; on the MS-COCO dataset, the rSum index reaches 544.9, which is significantly higher than the suboptimal method.

[0246] In addition, such as Figure 4 As shown, the method of the present invention has a faster training loss convergence speed than other existing methods and a lower loss value under the same training rounds.

[0247] Experimental results validate the effectiveness of the linguistically guided multi-level collaborative selection mechanism proposed in this invention, demonstrating that by integrating linguistic information such as part-of-speech and word frequency to guide the joint screening of regions and feature dimensions, the accuracy and robustness of image-text matching can be effectively improved.

[0248] As one implementation scheme, Figure 5 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0249] like Figure 5As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0250] Those skilled in the art will understand that Figure 5 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0251] like Figure 5 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0252] exist Figure 5 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0253] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0254] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0255] S10, preprocess and encode the input image and input text respectively to obtain the image region feature vector set and the text word-level feature sequence;

[0256] S20, generate word importance weight values ​​for each word in the input text based on the text word-level feature sequence, wherein the word importance weight values ​​are obtained by weighting a preset part-of-speech prior weight, inverse document frequency weight, and multilayer perceptron network weights generated based on the text word-level feature sequence;

[0257] S30, the text word-level feature sequence is weighted and aggregated according to the word importance weight value and mapped to the visual space to generate text global features, and the image global features corresponding to the image region feature vector set are calculated. The text global features are concatenated with the image global features and the image region feature vector set respectively, and the joint importance tensor is obtained after learning through a gating network.

[0258] S40, based on the joint importance tensor and the word importance weight value, filter the enhanced image features in the image region feature vector set;

[0259] S50, the text word-level feature sequence is segmented, fused, and updated according to the enhanced image features to obtain the enhanced text word-level feature sequence;

[0260] S60, construct an image-text similarity matrix based on the enhanced text word-level feature sequence and the image region feature vector set, and perform image-text matching based on the image-text similarity matrix.

[0261] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0262] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the linguistically guided multi-level collaborative selection image-text matching method described in the above embodiments.

[0263] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0264] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0265] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0266] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0267] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0268] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0269] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0270] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A linguistically guided multi-level collaborative selection image-text matching method, characterized in that, The method includes the following steps: S10, preprocess and encode the input image and input text respectively to obtain the image region feature vector set and the text word-level feature sequence; S20, generate word importance weight values ​​for each word in the input text based on the text word-level feature sequence, wherein the word importance weight values ​​are obtained by weighting a preset part-of-speech prior weight, inverse document frequency weight, and multilayer perceptron network weights generated based on the text word-level feature sequence; S30, the text word-level feature sequence is weighted and aggregated according to the word importance weight value and mapped to the visual space to generate text global features. Then, the image global features corresponding to the image region feature vector set are calculated. The text global features are concatenated with the image global features and the image region feature vector set respectively, and the joint importance tensor is obtained after learning through a gating network. S40, based on the joint importance tensor and the word importance weight value, filter the enhanced image features in the image region feature vector set; S50, the text word-level feature sequence is segmented, fused, and updated according to the enhanced image features to obtain the enhanced text word-level feature sequence; S60, construct an image-text similarity matrix based on the enhanced text word-level feature sequence and the image region feature vector set, and perform image-text matching based on the image-text similarity matrix; S20 specifically includes: S21, based on the word identifiers in the text word-level feature sequence, extract word vector features, part-of-speech embedding features, absolute position embedding features and relative position embedding features respectively, and concatenate the four types of features to obtain a multi-source linguistic feature sequence; Specifically, the part-of-speech embedding feature is obtained by acquiring the part-of-speech tags corresponding to each word through a pre-constructed part-of-speech dictionary and embedding them into a fixed dimension; the absolute position embedding feature is obtained by position encoding based on the absolute position index of the word in the sentence; and the relative position embedding feature is obtained by encoding based on the offset of the word relative to the center position of the sentence. S22, the multi-source linguistic feature sequence is input into a bidirectional gated recurrent unit network for bidirectional context encoding to obtain a context-aware linguistic feature representation; S23, apply a multi-head self-attention mechanism to model long-distance dependencies on the context-aware linguistic feature representation, and perform residual connections with the original features to obtain an enhanced linguistic feature representation. ; S24, the enhanced linguistic feature representation is input into a multilayer perceptron network, and the multilayer perceptron network weights of each word are obtained. : ； In the formula: Represents a multilayer perceptron network; S25, obtain the first Part-of-speech prior weights of each word : Among them, the part-of-speech prior weight For the first The preset importance weight associated with the part-of-speech category of each word in the pre-constructed part-of-speech dictionary; And, calculate the first Inverse document frequency weight of each word : ； In the formula: The total number of documents in the training corpus; For including the first The number of documents per word; S26, based on the weights of the multilayer perceptron network Part-of-speech prior weights and inverse document frequency weights Calculate the first Word importance weight value : ； in: , , These are the first, second, and third fusion coefficients that can be learned. The first, second, and third fusion coefficients are adaptively updated through backpropagation during training, and normalization constraints are applied before fusion to ensure that each fusion coefficient is non-negative and that the sum is 1.

2. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 1, characterized in that, In step S10, the preprocessing encoding includes: S11, the input text is segmented and indexed to obtain the word identifier sequence and corresponding length information of the text, and the word identifier sequence is padded to align to a preset maximum length; S12, embed the word identifier sequence after padding into word vectors, and perform context modeling on the embedded sequence based on a context encoding network to obtain the text word-level feature sequence; S13, determine the candidate salient regions in the input image, extract visual features from each salient region and map them to a preset feature dimension to obtain a preset number of image region feature vectors.

3. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 1, characterized in that, The expression for the joint importance tensor is: ； In the formula, Represents the joint importance tensor; This means that the features of each image region are concatenated with the global features of the text and then input into the spatial gating network, and the output is the first... The spatial importance of each region; This means that the global features of the image and the global features of the text are concatenated and then input into the channel-gated network, and the output is the first... The importance of each feature in the channel; in: ； ； In the formula, Represents global features of the text; Represents global features of an image; For the first Feature vectors of each image region; Indicates feature splicing; For channel gating network; It is the Sigmoid activation function. For spatial gating networks.

4. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 3, characterized in that, The text global features The calculation expression is: ； In the formula, For the first Textual features of each word; Sentence length; For linear projection layers; Indicates the first Each word's importance weight; The global features of the image The calculation expression is: ； In the formula, This represents the total number of image regions.

5. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 1, characterized in that, S40 includes: S41, normalize the word importance weights into a probability distribution. Calculate the probability distribution Information entropy, and use it as semantic complexity. Among them, the larger the information entropy value, the more complex the semantics of the text: ； ； S42, normalize the semantic complexity to obtain and according to Determine the number of regions to retain. Number of features retained : ； ； ； In the formula: and These are the minimum and maximum entropy values ​​within the current batch, respectively. and These represent the maximum and minimum number of regions that can be retained in the region dimension, respectively. and These represent the maximum and minimum number of feature dimensions that can be retained, respectively. Indicates rounding down; S43, averaging the joint importance tensor along the feature dimension to obtain the first... The overall importance weight of each region Select the one with the highest weight One region is assigned a weight of 1.0, and the remaining regions are assigned a decay coefficient. : ； ； In the formula: For area soft mask; The attenuation coefficient; For feature dimensions; S44, averaging the joint importance tensor along the region dimension to obtain the first... The combined importance weight of each feature dimension Select the one with the highest weight One feature dimension is assigned a weight of 1.0, and the remaining dimensions are assigned a decay coefficient. : ； ； In the formula: For feature dimension soft mask; S45, apply a soft mask to the region. With feature dimension soft mask Combined application to image region features yields enhanced image features after collaborative filtering. : ； In the formula: The first feature in the original image region The first region The values ​​of each feature dimension.

6. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 1, characterized in that, The S50 includes; S51, Calculate the self-similarity matrix between image regions based on enhanced image features. and text-image cross-modal similarity matrix : ； In the formula, This is the feature matrix of the original image region; This is the enhanced image feature matrix after collaborative filtering; This is the text word-level feature matrix; S52, perform learnable projection transformations on the self-similarity matrix and the cross-modal similarity matrix respectively, and perform matrix multiplication to obtain the cross-modal structural propagation features. : ； In the formula: For learnable projection transformation; S53, calculate and standardize the statistics of the cross-modal structural propagation features, and classify the word-level features into three categories: stable region, boundary region, and noise region based on the absolute value of the standardization result: ； in, For the stable region, For the boundary area, This is a noisy area; In the formula, The mean of the cross-modal structural propagation characteristics; The standard deviation of the cross-modal structure propagation characteristics; S54, the cross-modal structural propagation features and the text word-level feature sequence are weighted and fused using different fusion coefficients for the three types of regions to obtain the enhanced text word-level feature sequence. : ； In the formula, This is the text word-level feature matrix; , , These are the fusion coefficients for the three types of regions; and The threshold values ​​for dividing the first region and the threshold values ​​for dividing the second region are used.

7. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 1, characterized in that, The S60 includes: S61, calculate the... The text and the first Word-region attention matrix for each image : ； In the formula: No. Enhanced text word-level feature sequences of a text; For the first Features of an image; S62, averaging the attention matrix by word dimension to obtain the [number]th [word]. The text and the first Similarity between images : ； In the formula: Attention matrix The Row vector, representing the first row vector. The first text The word for the first Attention weight vectors for all regions of an image; For the first The length of each text; This is an attention aggregation function that includes weighted positive and negative similarity. S63, sort the candidate images or candidate texts according to the similarity and output the image-text matching results.

8. The linguistically guided multi-level collaborative selection image-text matching method as described in claim 7, characterized in that, Following S60, the following is also included: The image-text matching results are optimized using a bidirectional ranking loss function, the expression of which is: ； in: ； ； In the formula, For image-to-text retrieval loss; For text-to-image retrieval loss; These are boundary hyperparameters; Indicates the first The similarity between an image and its corresponding matching text. Indicates the first The text and the first Similarity between images.

9. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the linguistically guided multi-level collaborative selection image-text matching method as described in any one of claims 1 to 8.

Images