Image-text alignment model training method and device, computer equipment and storage medium

Abstract

The invention discloses an image-text alignment model training method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding of input image data and text data, and obtaining an image block feature sequence and a text token feature sequence; performing bidirectional feature interaction on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features; inputting the fused image features, the fused text prompt features and a preset output token into an SAM mask decoder to generate an image mask; filtering the fused image features based on the image mask, extracting image global features related to the text description, and obtaining text global features; constructing a joint loss function; and performing end-to-end training on the model based on the joint loss function until the model converges. According to the method, the limitation of coarse-grained alignment of a traditional model is effectively solved, and the pertinence and matching precision of image-text features are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image-text information recognition, and more particularly to a training method and device of an image-text alignment model, a computer device and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, cross-modal learning has become a core research direction in the field of computer vision and natural language processing. As a key bridge connecting images and natural language, image-text alignment technology has shown wide application value in image retrieval, text generation, visual question answering and other downstream tasks.

[0003] To achieve effective alignment of image-text semantics, various solutions have been proposed in the prior art, among which the CLIP (Contrastive Language-Image Pretraining) model proposed by OpenAI is representative. This model jointly trains image encoders and text encoders on large-scale image-text pairs through a contrastive learning mechanism, encodes images and texts into fixed-dimensional feature vectors, and realizes semantic alignment in a shared embedding space. Specifically, the image encoder of CLIP often uses a convolutional neural network or a visual Transformer architecture to convert images into image feature vectors; the text encoder encodes natural language texts into text feature vectors based on the Transformer architecture, and the two form a unified semantic space through training. In the training process, CLIP uses a many-to-one contrastive loss function, taking image-text pairs in the training batch as positive samples and other combinations as negative samples, to maximize the similarity of positive samples and minimize the similarity of negative samples to achieve feature alignment; its training data relies on large-scale web scraped image-text pairs, without the need for manually labeled class labels, giving the model open vocabulary recognition ability and zero-shot classification generalization ability, significantly promoting the development of image-text understanding and retrieval technology.

[0004] However, the existing CLIP model still has obvious limitations in image-text alignment: it mainly performs coarse-grained matching through the embedding vectors of image global features and entire text features, lacking the ability to capture fine-grained semantic associations between image local regions (such as objects, attributes, actions, etc.) and text segments (such as words, phrases). In practical applications, this coarse-grained alignment mechanism is difficult to meet the needs of complex scenarios. For example, in the image-text retrieval task, when the input contains text describing local features (such as "person with yellow hair"), the model may return results containing irrelevant global features (such as "person with black hair next to a yellow vehicle") due to the inability to accurately associate text details with image local regions, resulting in insufficient matching accuracy and poor performance in downstream tasks.

[0005] Therefore, how to break through the limitation of the existing coarse-grained alignment, realize the accurate correspondence between the local features of the image and the details of the text, and improve the fine-grained level of the image-text semantic understanding has become a technical problem to be solved in the field. SUMMARY

[0006] The present application aims to overcome the deficiencies of the prior art and provide a training method and device for an image-text alignment model, a computer device and a storage medium.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a training method for an image-text alignment model, comprising:

[0009] The input image data and text data are respectively encoded to obtain an image block feature sequence and a text token feature sequence, and the label of the image data includes a text description of the image and a label mask related to the description;

[0010] The image block feature sequence and the text token feature sequence are subjected to bidirectional feature interaction to obtain fused image features and fused text prompt features;

[0011] The fused image features, the fused text prompt features and a preset output token are input into a SAM mask decoder to generate an image mask;

[0012] The fused image features are filtered based on the image mask to extract image global features related to the text description;

[0013] Text global features are extracted from the text token feature sequence;

[0014] A contrast loss is calculated based on the image global features and the text global features, and a joint loss function is constructed by combining a mask loss of the image mask and the label mask;

[0015] The model is trained end-to-end based on the joint loss function until the model converges.

[0016] Further, the encoding of the input image data and text data to obtain the image block feature sequence and the text token feature sequence comprises:

[0017] The image is divided into a plurality of image blocks according to a preset size, each image block is subjected to linear projection and position encoding to obtain initial image block features;

[0018] The initial image block features are subjected to multi-layer attention calculation and perception machine processing to output an image block feature sequence;

[0019] The text is segmented, the segmentation result is converted into a token embedding, and position encoding is added to obtain initial text token features;

[0020] The initial text token features are subjected to multi-layer attention calculation and perception machine processing to output a text token feature sequence.

[0021] Further, the bidirectional feature interaction between the image block feature sequence and the text token feature sequence to obtain the fused image feature and the fused text prompt feature comprises:

[0022] The text token feature sequence is projected into a prompt query vector, a prompt key vector, and a prompt value vector;

[0023] Based on the prompt query vector, text-to-image cross-attention calculation is performed on the image block feature sequence to update the text prompt feature;

[0024] Based on the updated key vector and value vector of the text prompt feature, image-to-text cross-attention calculation is performed on the image block feature sequence to update the image block feature sequence;

[0025] The text prompt feature and the updated image block feature sequence are updated according to a preset number of times to obtain the fused image feature and the fused text prompt feature.

[0026] Further, the fused image feature, the fused text prompt feature, and the preset output token are input into the SAM mask decoder to generate an image mask, comprising:

[0027] The fused image feature is upsampled to obtain a high-resolution image feature map matching the original image spatial resolution;

[0028] The preset output token is mapped to a dynamic linear classifier through a multi-layer perception machine;

[0029] The high-resolution image feature map is input into the dynamic linear classifier to calculate the mask foreground probability of each spatial position in the image, and the image mask is generated based on the foreground probability.

[0030] Further, the fused image feature is filtered based on the image mask to extract image global features related to the text description, comprising:

[0031] The fused image feature is multiplied by the image mask to obtain a multiplication result;

[0032] The multiplication result is averaged as the image global feature.

[0033] Further, the text global feature is extracted from the text token feature sequence, and the extraction includes:

[0034] Any one of the following features is selected from the output last layer text token feature sequence as the text global feature, and the any one of the following features includes: an embedding vector of a first token in the text token feature sequence, an embedding vector of a token used to identify the end of the text in the text token feature sequence, and a vector obtained by averaging embedding vectors of all tokens in the text token feature sequence.

[0035] Further, the contrastive loss is a bidirectional InfoNCE loss, which includes an image-to-text contrastive loss and a text-to-image contrastive loss, the mask loss is a Dice loss, and the joint loss function is a weighted sum of the contrastive loss and the mask loss.

[0036] In a second aspect, the present application further provides a training device of a text-image alignment model, which includes:

[0037] An encoding unit is configured to encode input image data and text data respectively to obtain an image block feature sequence and a text token feature sequence, and a label of the image data includes a text description of the image and a label mask related to the description;

[0038] A bidirectional fusion unit is configured to perform bidirectional feature interaction on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features;

[0039] An image mask generation unit is configured to input the fused image features, the fused text prompt features, and a preset output token into a SAM mask decoder to generate an image mask;

[0040] A filtering unit is configured to filter the fused image features based on the image mask to extract image global features related to the text description;

[0041] An extraction unit is configured to extract a text global feature from the text token feature sequence;

[0042] A construction unit is configured to calculate a contrastive loss based on the image global features and the text global features, and construct a joint loss function in combination with a mask loss of the image mask and the label mask;

[0043] A training unit is configured to perform end-to-end training on the model based on the joint loss function until the model converges.

[0044] In a third aspect, the present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method of the image-text alignment model when executing the computer program.

[0045] In a fourth aspect, the present application also provides a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program comprises program instructions, which, when executed by a processor, cause the processor to execute the training method of the image-text alignment model.

[0046] Compared with the prior art, the training method of the image-text alignment model has the following beneficial effects: the training method of the image-text alignment model comprises: respectively encoding input image data and text data to obtain an image block feature sequence and a text token feature sequence, wherein the label of the image data comprises a text description of the image and a label mask related to the description; performing bidirectional feature interaction on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features; inputting the fused image features, the fused text prompt features, and a preset output token into a SAM mask decoder to generate an image mask; filtering the fused image features based on the image mask to extract image global features related to the text description; extracting text global features from the text token feature sequence; calculating a contrast loss based on the image global features and the text global features, and constructing a joint loss function by combining a mask loss of the image mask and the label mask; and performing end-to-end training on the model based on the joint loss function until the model converges. The present application realizes bidirectional interaction of image and text features through image-text embedding fusion, filters irrelevant image features by means of the SAM mask decoder, and strengthens training supervision by combining the joint loss function of the contrast loss and the Dice loss, thereby effectively solving the limitations of traditional model coarse-grained alignment, accurately capturing the corresponding relationship between image local regions and text segments, improving the pertinence and matching accuracy of image-text features, and enabling the model to accurately respond to queries containing local feature descriptions in image-text retrieval and other downstream tasks, thereby significantly improving the performance of downstream tasks.

[0047] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purpose characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the training method for the image-text alignment model provided in a specific embodiment of the present invention;

[0050] Figure 2 A schematic block diagram of a training device for a text-image alignment model provided in a specific embodiment of the present invention;

[0051] Figure 3 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0054] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0056] like Figure 1 As shown, this embodiment of the invention provides a training method for a text-image alignment model, including the following steps: S10-S70.

[0057] S10, encode the input image data and text data respectively to obtain image block feature sequences and text token feature sequences, and the label of the image data includes a text description of the image and a label mask related to the description.

[0058] Step S10 is used for feature encoding of the input image data and text data, and the original image and text are respectively converted into structured feature sequences (image block feature sequences and text token feature sequences) by an image encoder and a text encoder, to provide basic feature support for subsequent bidirectional feature interaction and fine-grained alignment. The label of the image data includes a text description (such as “a black cat sitting on a sofa”) corresponding to the image content and a label mask (such as a mask marking the “black cat” area in the image) related to the description, and the label mask is used for subsequent calculation and supervision of the mask loss.

[0059] In an embodiment, step S10 specifically includes the following: S101-S104.

[0060] S101, divide the image into a plurality of image blocks according to a preset size, perform linear projection on each image block, and add position encoding to obtain initial image block features.

[0061] Image block division: the size of the input original image is HxWx3 (where H is the image height, W is the image width, and 3 is the number of RGB channels), and the image is uniformly divided into N non-overlapping image blocks according to a preset size PxP, where N = HxW / (PxP). For example, the size of the original image is 224x224x3, the preset image block size P = 16, and the number of image blocks N = 224x224 / (16x16) = 196.

[0062] Each P x P x 3 image block is flattened into a one-dimensional vector with a length of 3 x P x P (for example, a 16 x 16 x 3 image block is flattened to a length of 768), and a linear projection layer is used to map the vector to a preset dimension D (for example, D = 768) to obtain an initial feature vector of the image block.

[0063] In order to retain the spatial position information of the image block, a learnable 1D position encoding (the dimension of the position encoding is consistent with the dimension of the image block feature vector, both being D) is added to each linearly projected image block feature vector, and a learnable special marker [CLS] is spliced at the first position of all image block feature sequences (for subsequent feature aggregation reference), and finally the initial image block features are obtained.

[0064] S102, perform multi-layer attention calculation and perception machine processing on the initial image block features to output image block feature sequences.

[0065] The initial image block features (including [CLS] token and position encoding) are input into a stacked Transformer encoder, which contains L layers of standard "Multi-head Self-Attention + Multi-Layer Perceptron (MLP)" structure. For example, a Transformer encoder with ViT-Base architecture is used, where L = 12 (number of layers), D = 768 (feature dimension), and the number of heads h = 12.

[0066] The processing logic of each layer of the Transformer is as follows.

[0067] Multi-head self-attention layer: multi-dimensional attention calculation is performed on the input features to capture the correlation between different image blocks (such as the correlation between the "cat's head" and "cat's body" image blocks).

[0068] MLP layer: non-linear transformation is performed on the features after attention calculation (such as through two fully connected networks + GELU activation function).

[0069] Layer normalization: layer normalization is added before and after the multi-head attention layer and the MLP layer to stabilize the training process.

[0070] After processing by the L-layer Transformer encoder, a sequence of image block features containing spatial correlation information is output.

[0071] S103, perform word segmentation processing on the text, convert the word segmentation results into token embeddings and add position encoding to obtain initial text token features.

[0072] Word segmentation: a pre-set word segmentation tool is used to perform word segmentation on the input text (such as "a black cat sitting on the sofa"), resulting in T text tokens (such as the word segmentation result is "one / only / sit / in / sofa / on / of / black / cat", T = 9). For example, the word segmentation tool uses the Byte-Pair Encoding (BPE) word segmentation method consistent with CLIP, which supports open-vocabulary token division.

[0073] Token embedding: each text token is mapped to an embedding vector with dimension D (the embedding vector is initialized by pre-training or randomly and learned during training), where D is consistent with the feature dimension of the image encoder (such as D = 768), ensuring that the dimensions of the subsequent feature interactions are matched.

[0074] Position encoding addition: a learnable 1D position encoding (dimension D) is added to each token embedding vector to preserve the temporal order information of the text tokens (such as the order relationship between "black" and "cat"), and finally the initial text token features are obtained.

[0075] S104, multi-layer attention calculation and perception machine processing are performed on the initial text token features, and a text token feature sequence is output.

[0076] The initial text token features are processed by a text Transformer encoder for context association, and the specific process is as follows:

[0077] The obtained initial text token features are input into a stacked text Transformer encoder, which adopts a similar "multi-layer attention + MLP" architecture as the image Transformer and contains L layers of Transformer modules. For example, the parameter settings of the text Transformer are L = 12 (number of layers), D = 512 (feature dimension), and the number of multi-head attention heads is 8 1-52.

[0078] The processing logic of each layer of the text Transformer is as follows:

[0079] Multi-head self-attention layer: captures the context association between text tokens (such as the modifying relationship between "black" and "cat", and the action-scene relationship between "sit" and "sofa").

[0080] MLP layer: performs nonlinear transformation on the attention features to enhance the feature expression capability.

[0081] Layer normalization: stabilizes the training gradient and improves the model convergence speed.

[0082] After L layers of processing, a text token feature sequence containing context semantic information is output.

[0083] For step S10, the image encoding process preserves the spatial structure information of the original image through image block division and position encoding, and combines the multi-head attention mechanism of the Transformer to capture the association between image blocks, laying a foundation for subsequent extraction of local region features.

[0084] S20, bidirectional feature interaction is performed on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features.

[0085] Step S20 realizes the bidirectional fine-grained interaction of the image block feature sequence and the text token feature sequence through the image-text embedding fusion module, so that the image features carry the text semantic guidance information and the text features fuse the visual context details, thereby providing fusion feature support for subsequent mask generation and fine-grained alignment. This process is designed with reference to PromptEncoder and Two-WayTransformer in SAM (Segment Anything Model), and through the bidirectional attention interaction of text prompts and image features, multi-level semantic association capture is realized.

[0086] In an embodiment, step S20 specifically includes steps S201-S204.

[0087] S201, projecting the text token feature sequence into a prompt query vector, a prompt key vector, and a prompt value vector.

[0088] The input text token feature sequence is projected into vectors through PromptEncoder. The PromptEncoder includes 1-2 layers of small multi-layer perceptron (MLP), each layer adopts a linear transformation + activation function (such as GELU) structure.

[0089] For example, the text token feature sequence “a black cat sitting on a sofa” corresponds to T=9 token features (dimension D=768), and after 1 layer MLP (input dimension 768, output dimension 768, containing GELU activation) processing, three groups of vectors are generated:

[0090] Prompt query vector: used to guide the attention calculation of image features.

[0091] Prompt key vector: used for image-to-text attention interaction.

[0092] Prompt value vector: used for image-to-text feature update.

[0093] The dimensions of the three groups of vectors after projection are consistent with the dimension of the input text token features (D=768), ensuring matching with the dimension of the image features.

[0094] S202, performing cross-attention calculation based on the prompt query vector on the image block feature sequence to update the text prompt feature.

[0095] The cross-attention calculation logic is as follows:

[0096] Calculate attention weight: generate an attention weight matrix through dot product operation of the prompt query vector and the image block feature.

[0097] updating the text prompt feature: based on the attention weight matrix, the image block feature sequence is weighted and summed to obtain the updated text prompt feature value fused with image information, and the text prompt feature is updated through residual connection.

[0098] For example, the text prompt query vector "black" "cat" corresponds to the prompt query vector that gives higher weight to the image block of the cat's black hair region in the image block feature, so that the updated text prompt feature carries the visual feature information of the cat.

[0099] S203, based on the key vector and value vector of the updated text prompt feature, the image block feature sequence is calculated by cross-attention from image to text, and the image block feature sequence is updated.

[0100] Based on the key vector and value vector of the updated text prompt feature in step S202, the image block feature sequence is calculated by cross-attention from image to text. The cross-attention calculation logic is:

[0101] Calculate attention weight: take image block feature as query vector, and perform dot product operation with updated text prompt key vector to generate attention weight matrix.

[0102] Update image block feature: based on the attention weight matrix, the text prompt value vector is weighted and summed to obtain the updated image block feature value fused with text information, and the image block feature sequence is updated through residual connection.

[0103] For example, the head region of the cat in the image block feature will give higher weight to the text prompt key vector corresponding to the text prompt "cat" "black", so that the updated image block feature carries the semantic label information of "cat".

[0104] S204, update the text prompt feature and update the image block feature sequence processing according to the preset number of times to obtain the fused image feature and the fused text prompt feature.

[0105] Repeat steps S202 (text-to-image cross-attention) and S203 (image-to-text cross-attention) for a preset number of times M (such as M=2), and after each repetition, the updated text prompt feature and image block feature are respectively processed by multilayer perceptron (MLP) and layer normalization (LayerNorm) operation.

[0106] For example, M=2:

[0107] After the first round of interaction: the text prompt feature fuses the preliminary image region information, and the image block feature fuses the preliminary text semantic information;

[0108] After the second round of interaction: through the calculation of attention again, the text prompt feature is more accurate to the image details (such as "black" corresponding to the cat's hair area feature), and the image block feature is more accurate to the text description (such as the image block feature of the cat strengthening the semantics of "cat" and "sit");

[0109] Layer normalization: after each interaction, the features are normalized by layer (calculate the mean and variance of the features, and standardize them to N(0, 1) distribution), to stabilize the training process;

[0110] MLP processing: through a 1-layer MLP (such as input and output dimensions are both D=768, containing a GELU activation) to perform nonlinear transformation on the features, to enhance the feature expression ability.

[0111] After M rounds of interaction, the fused image features (updated image block feature sequence and fused text prompt feature) are output.

[0112] For step S20, through the bidirectional cross-attention mechanism, the text prompt guides the image features to focus on the relevant areas, and the image features feed back to the text prompt to strengthen the visual semantics, solving the problem of fuzzy semantic association in the global alignment of traditional models; at the same time, through the position encoding and attention weight transmission in the interaction process, the spatial position relationship of the image block and the time sequence logical relationship of the text token are completely preserved, ensuring the structural integrity of the fused features; in addition, the combination of multiple rounds of interaction with MLP and layer normalization makes the fused features not only capture low-level visual details (such as color, shape), but also associate high-level semantic information (such as object category, attribute), providing high-quality input for subsequent mask generation and global feature extraction.

[0113] S30, input the fused image features, the fused text prompt features and the preset output token into the SAM mask decoder to generate an image mask.

[0114] Step S30 realizes accurate positioning of key areas in the image. This process restores the feature space resolution through upsampling, adapts to the semantic requirements through a dynamic classifier, and finally outputs a mask that can be used to filter irrelevant features, providing spatial guidance for subsequent fine-grained feature extraction.

[0115] In an embodiment, step S30 specifically includes the following steps: S301-S303.

[0116] S301, perform upsampling processing on the fused image features to obtain a high-resolution image feature mapping matching the original image spatial resolution.

[0117] The fused image feature (i.e., the updated image block feature sequence) is input. The feature is resolution-enhanced by upsampling, which can use transposed convolution or bilinear interpolation, and the upsampling multiple is P (corresponding to the image block division size).

[0118] For example, the original image size is 224x224, the image block size P=16, and the initial spatial size of the fused image feature is 14x14 (224 / 16=14); 2 layers of transposed convolution are used for upsampling (the first layer of transposed convolution kernel size 3x3, step size 2, output channel D / 2; the second layer of transposed convolution kernel size 3x3, step size 8, output channel D / 4), and finally a high-resolution image feature mapping matching the spatial resolution of the original image is obtained, with a size of 224x224 and a feature dimension of D / 4 (e.g., when D=768, the output dimension is 192).

[0119] S302, mapping the preset output token to a dynamic linear classifier by a multi-layer perception.

[0120] The preset output token is a special learnable vector used to guide mask generation, and its initial value is randomly initialized and learned through backpropagation during the training process, and is used to encode the global instruction of "generating a text-related mask".

[0121] The preset output token is mapped by a multi-layer perception (MLP): the MLP includes 2-3 layers of linear transformation, each layer uses a GELU activation function, the input dimension is D, and the output dimension is CxDfeat (where C is the number of classifiers, usually C=1; Dfeat is the dimension of the high-resolution image feature mapping, such as 192). After mapping, a dynamic linear classifier and bias are obtained, and the parameters of the classifier are dynamically adjusted with the preset output token and the training process, which can adapt to the semantic requirements of different text descriptions (such as "yellow hair" and "black vehicle" corresponding to different classifier parameters).

[0122] For example, after the preset output token t is mapped by 2 layers of MLP (the first layer input 768, output 384, and the second layer input 384, output 192), a dynamic linear classifier and bias are obtained, which are used for subsequent foreground probability calculation.

[0123] S303, inputting the high-resolution image feature mapping into the dynamic linear classifier to calculate the mask foreground probability of each spatial position in the image, and generating an image mask based on the foreground probability.

[0124] The obtained high-resolution image feature map is operated with the obtained dynamic linear classifier W and bias b, a feature vector of each spatial position is calculated by a formula, and a mask foreground probability of the position is obtained through a sigmoid activation function, and the value is closer to 1, indicating that the position is more relevant to the text description.

[0125] A probability threshold is set, positions of the foreground probability are marked as foreground (value 1), and the remaining positions are marked as background (value 0), to generate a binary image mask.

[0126] For example, when the text description is “person with yellow hair”, the feature vector of the hair region of the person in the high-resolution feature map has a higher score after operation with the dynamic classifier, and the foreground probability is close to 1 after sigmoid activation. The generated mask M marks the yellow hair region as foreground, and the remaining regions are marked as background, thereby realizing accurate positioning of the key region.

[0127] S40, filtering the fused image features based on the image mask to extract image global features related to the text description.

[0128] Step S40 realizes accurate expression of image semantics. The process removes irrelevant region interference through product operation of the mask and the features, and then generates global features focusing on the text key information through mean aggregation, thereby providing high-quality image feature input for subsequent image-text comparison loss calculation.

[0129] In an embodiment, step S40 specifically includes the following steps: S401-S402.

[0130] S401, performing product operation on the fused image features and the image mask to obtain a product operation result.

[0131] First, the image mask is adapted: the binary mask M is down-sampled according to the image block division size (P x P) to obtain a mask vector matching the dimension of the image block feature sequence. The down-sampling method adopts maximum pooling, that is, if there is at least one foreground pixel (mask value 1) in the original image region corresponding to the image block, the mask value of the image block = 1.

[0132] For example, the original image size is 224 x 224, the image block size is 16 x 16, the number of image blocks N = 196, and the image mask M is a binary matrix marking the “yellow hair region”; after down-sampling, the image block element value corresponding to the hair region in the mask vector m is 1, and the image block element value of the remaining regions (such as background and other parts of the body) is 0.

[0133] Product operation is performed: the fused image block features are multiplied by the corresponding mask values element by element to obtain filtered image block features. If the image block is irrelevant to the text description, its features are cleared; if relevant, the features are completely retained, effectively filtering irrelevant features.

[0134] S402, average the product operation result as the image global feature.

[0135] The filtered image block feature sequence obtained in step S401 is averaged to obtain the image global feature, which is used to normalize the feature scale and avoid feature deviation caused by size differences of relevant regions. If all image blocks are irrelevant, the global feature is set to a zero vector.

[0136] For example, the text description is "person with yellow hair", the filtered image block feature sequence only retains the 10 image block features of the hair region, and the global feature obtained by aggregating these features will focus on the visual features (such as color, texture) of "yellow hair", rather than the features of the background or other irrelevant regions.

[0137] For step S40, through the product operation of the mask and the feature, the region features in the image irrelevant to the text description (such as the background, non-target objects) are effectively filtered, solving the problem of traditional global features containing redundant information, making the image features more focused on the key semantics of the text; at the same time, the aggregation process only calculates the mean value based on the relevant image block features, retaining the local detailed features (such as color, shape, attribute) corresponding to the text description, improving the expression ability of the image global feature to fine-grained semantics; in addition, the dynamic generation of the mask makes the feature filtering process flexible with the text description (such as "yellow hair" and "red clothes" corresponding to different filtering regions), enhancing the adaptability of the model to complex scenes; the global feature focusing on the text-related region and the text global feature are more easily aligned in subsequent contrast learning, laying the foundation for improving the image-text matching accuracy.

[0138] S50, obtain the text global feature from the text token feature sequence.

[0139] Specifically, the way to obtain the text global feature is: select any one of the following features from the output last layer text token feature sequence as the text global feature, wherein any one of the following features includes:

[0140] Method one: the embedding vector of the first token in the text token feature sequence.

[0141] The first token in the token sequence of the text after being segmented usually is a special start marker (such as “[CLS]” or a sentence start symbol), or the first semantic token of the text. During the processing of the text encoder (such as a Transformer), the embedding vector of the first token aggregates the context information of the entire text sequence through a self-attention mechanism, and can be used as a condensed representation of the overall semantics of the text.

[0142] For example, the text “a black cat sitting on a sofa” is segmented to obtain the token sequence “[CLS], a, a, sits, on, a sofa, on, a, black, cat, [EOS]”, where the first token is “[CLS]”; the embedding vector corresponding to “[CLS]” is directly extracted from the last layer of the text token feature sequence as the global feature of the text. The vector has fused all semantic information of “black cat”, “sits”, “sofa”, etc. through multi-layer attention calculation, and can represent the overall meaning of the text.

[0143] Method two: embedding vector of a token in the text token feature sequence used to identify the end of the text.

[0144] The text token sequence usually contains a special token (such as “[EOS]”) used to identify the end of the text. The embedding vector of this token will capture the semantic association at the end of the text during the encoding process, and indirectly aggregate the information of all previous tokens, which is suitable as a global feature of the text.

[0145] For example, the token sequence of the above text ends with “[EOS]”, and the embedding vector corresponding to “[EOS]” is extracted from the last layer of the feature sequence as the global feature of the text. Since the self-attention mechanism of the Transformer encoder has global dependency characteristics, the feature vector of “[EOS]” already contains semantic associations of key tokens such as “black”, “cat”, “sits”, etc., which can effectively represent the overall description of the text.

[0146] Method three: vector obtained by averaging the embedding vectors of all tokens in the text token feature sequence.

[0147] By averaging the embedding vectors of all tokens in the text token feature sequence, the semantic information of all tokens can be integrated, avoiding the semantic bias that may exist in a single special token, and this method is particularly suitable for long text scenarios.

[0148] S60, based on the image global feature and the text global feature, calculate a contrastive loss, and combine a mask loss of the image mask and the label mask to construct a joint loss function.

[0149] The step S60 provides a comprehensive supervision signal for model training. Through the combination of bidirectional semantic alignment supervision and mask precision supervision, the process guides the model to simultaneously learn fine-grained image-text correlation and accurate region positioning capability, ultimately improving the fine-grained alignment performance of the model.

[0150] Specifically, the contrast loss is a bidirectional InfoNCE loss, which includes image-to-text contrast loss and text-to-image contrast loss, the mask loss is a Dice loss, and the joint loss function is the weighted sum of the contrast loss and the mask loss.

[0151] The contrast loss is used to supervise the alignment effect of image global features and text global features in the shared semantic space. A bidirectional InfoNCE loss (image-to-text loss + text-to-image loss) is adopted, and the specific calculation process is as follows:

[0152] I. Feature normalization

[0153] The image global feature g extracted in S40 I and the text global feature g extracted in step S50 T are subjected to L2 normalization processing to eliminate the influence of feature scale difference on similarity calculation. The normalization formula is:

[0154] where ||·||2 represents the L2 norm.

[0155] II. Similarity calculation

[0156] The semantic similarity of the normalized image global feature and the text global feature is calculated, and the formula is: where is the normalized global feature of the i-th image, is the normalized global feature of the j-th text, · represents vector dot product (cosine similarity), and τ is a learnable temperature parameter used to adjust the steepness of the similarity distribution.

[0157] III. Bidirectional InfoNCE loss calculation

[0158] For each image I i , the proportion of its similarity with the corresponding text T i in all texts is calculated, and the formula is:

[0159] For each text T j , the proportion of its similarity with the corresponding image I j in all images is calculated, and the formula is:

[0160]

[0161] Take the average of the two as the final contrast loss, that is:

[0162]

[0163] The mask loss is used to supervise the matching accuracy of the image mask generated by the model and the label mask in the training data, and the Dice loss is adopted, and the specific calculation process is as follows:

[0164] Suppose that the image mask generated by the model is M H×W ∈ {0, 1} GT , and the label mask (manually annotated region mask related to the text description) in the training data is M H×W ∈ {0, 1} align , and the Dice loss formula is:

[0165] Wherein, is a minimum value (such as =1e-5), to avoid zero denominator; The numerator is twice the intersection of the mask, and the denominator is the approximate union of the mask, and the loss value range is [0, 1], and the smaller the value is, the higher the overlap degree of the generated mask and the label mask is.

[0166] The joint loss function is the weighted sum of the contrast loss and the mask loss, and the formula is:

[0167] L = L align + λ·L mask

[0168] Wherein, λ is a weight coefficient, used to balance the importance of semantic alignment supervision and mask accuracy supervision, which can be dynamically adjusted according to the training effect.

[0169] S70, based on the joint loss function, the model is trained end to end until the model converges.

[0170] The application realizes the bidirectional interaction of image and text features through the fusion of image-text embedding, generates a mask to filter irrelevant image features by means of the SAM mask decoder, and strengthens the training supervision by combining the joint loss function of the contrast loss and the Dice loss, effectively solves the limitations of the traditional model coarse-grained alignment, accurately captures the corresponding relationship between the local region of the image and the text segment, improves the pertinence and matching accuracy of the image-text features, and enables the model to accurately respond to the query containing local feature description in the downstream tasks such as image-text retrieval, and significantly improves the performance of the downstream tasks.

[0171] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0172] The embodiment of the present application also provides a training device of a picture-text alignment model, which is used for executing the steps in any one of the foregoing embodiments of the training method of the picture-text alignment model. Specifically, refer to Figure 2 , Figure 2 A schematic block diagram of the training device 100 of the picture-text alignment model provided by the embodiment of the present application is shown, which specifically comprises:

[0173] The encoding unit 110 is configured to encode the input image data and text data respectively to obtain an image block feature sequence and a text token feature sequence, and the label of the image data comprises a text description of the image and a label mask related to the description; the bidirectional fusion unit 120 is configured to perform bidirectional feature interaction on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features; the image mask generation unit 130 is configured to input the fused image features, the fused text prompt features and a preset output token into a SAM mask decoder to generate an image mask; the filtering unit 140 is configured to filter the fused image features based on the image mask to extract image global features related to the text description; the extraction unit 150 is configured to extract text global features from the text token feature sequence; the construction unit 160 is configured to calculate a contrastive loss based on the image global features and the text global features, and construct a joint loss function in combination with a mask loss of the image mask and the label mask; and the training unit 170 is configured to perform end-to-end training on the model based on the joint loss function until the model converges.

[0174] In an embodiment, the encoding unit 110 is specifically configured to: divide the image into a plurality of image blocks according to a preset size, perform linear projection on each image block and add position encoding to obtain initial image block features; perform multi-layer attention calculation and a perception machine processing on the initial image block features to output an image block feature sequence; perform word segmentation processing on the text, convert the word segmentation result into token embedding and add position encoding to obtain initial text token features; and perform multi-layer attention calculation and a perception machine processing on the initial text token features to output a text token feature sequence.

[0175] In an embodiment, the bidirectional fusion unit 120 is specifically configured to: project the text token feature sequence into a prompt query vector, a prompt key vector and a prompt value vector; perform text-to-image cross-attention calculation on the image block feature sequence based on the prompt query vector to update the text prompt features; perform image-to-text cross-attention calculation on the image block feature sequence based on the key vector and the value vector of the updated text prompt features to update the image block feature sequence; and update the text prompt features and the updated image block feature sequence for a preset number of times to obtain the fused image features and the fused text prompt features.

[0176] In an embodiment, the image mask generation unit 130 is specifically configured to: perform up-sampling processing on the fused image features to obtain a high-resolution image feature map matching the original image spatial resolution; map a preset output token to a dynamic linear classifier through a multi-layer perception; input the high-resolution image feature map into the dynamic linear classifier to calculate a mask foreground probability of each spatial position in the image, and generate an image mask based on the foreground probability.

[0177] In an embodiment, the filtering unit 140 is specifically configured to: perform a product operation on the fused image features and the image mask to obtain a product operation result; and take an average value of the product operation result as an image global feature.

[0178] In an embodiment, the extraction unit 150 is specifically configured to: select any one of the following features from the output last layer text token feature sequence as a text global feature, wherein the any one of the following features includes: an embedding vector of a first token in the text token feature sequence, an embedding vector of a token used to identify the end of the text in the text token feature sequence, and a vector obtained by averaging all tokens in the text token feature sequence.

[0179] It should be noted that the specific implementation process of the training device 100 and each unit of the above image-text alignment model can be clearly understood by those skilled in the art, which can be referred to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0180] The training device of the image-text alignment model can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 3 .

[0181] Please refer to Figure 3 , Figure 3 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 700 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0182] As shown in Figure 3 , the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the image-text alignment model training method as described above when executing the computer program.

[0183] The computer device 700 includes a processor 720, a memory, and a network interface 750 connected through a system bus 710, wherein the memory can include a non-volatile storage medium 730 and an internal memory 740.

[0184] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. The computer program 732, when executed, can cause the processor 720 to perform the training method of the image-text alignment model.

[0185] The processor 720 is configured to provide computing and control capabilities to support the operation of the entire computer device 700.

[0186] The memory 740 provides an environment for the execution of the computer program 732 in the non-volatile storage medium 730. The computer program 732, when executed by the processor 720, can cause the processor 720 to perform the training method of the image-text alignment model.

[0187] The network interface 750 is configured to perform network communication, such as sending an assigned task, etc. Those skilled in the art can understand that Figure 3 The structure shown in FIG. 7 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 700 to which the scheme of the present application is applied. Specifically, the computer device 700 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. The processor 720 is configured to run the program code stored in the memory to implement the training method of the image-text alignment model.

[0188] Those skilled in the art can understand that Figure 3 The embodiment of the computer device shown in FIG. 7 does not constitute a limitation on the specific structure of the computer device. In other embodiments, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device can only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are consistent with those of the memory 740 and the processor 720 shown in the embodiment, and will not be described here. Figure 3

[0189] It should be understood that, in the embodiments of the present application, the processor 720 can be a central processing unit (CPU), and the processor 720 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.​

[0190] In another embodiment of the present application, a computer readable storage medium is provided. The computer readable storage medium can be a non-transitory computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the training method of the text-alignment model disclosed in the embodiments of the present application.

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in a general manner in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0192] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, or units with the same function can be combined into one unit, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0193] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or software functional unit.

[0195] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a text-image alignment model, characterized in that, The method comprises the following steps: encoding the input image data and text data respectively to obtain image block feature sequences and text token feature sequences, wherein the label of the image data comprises a text description of the image and a label mask related to the description; performing bidirectional feature interaction on the image block feature sequences and the text token feature sequences to obtain fused image features and fused text prompt features; inputting the fused image features, the fused text prompt features and a preset output token into a SAM mask decoder to generate an image mask; filtering the fused image features based on the image mask to extract image global features related to the text description; extracting text global features from the text token feature sequences; calculating a contrast loss based on the image global features and the text global features, and constructing a joint loss function by combining a mask loss of the image mask and the label mask; performing end-to-end training on the model based on the joint loss function until the model converges. 2.The method of claim 1, wherein, The method comprises the following steps: dividing the image into multiple image blocks according to a preset size, performing linear projection on each image block and adding position encoding to obtain initial image block features; performing multi-layer attention calculation and perception machine processing on the initial image block features to output image block feature sequences; performing word segmentation on the text, converting the word segmentation result into token embedding and adding position encoding to obtain initial text token features; performing multi-layer attention calculation and perception machine processing on the initial text token features to output text token feature sequences. 3.The method of claim 1, wherein, The method comprises the following steps: projecting the text token feature sequences into prompt query vectors, prompt key vectors and prompt value vectors; performing text-to-image cross-attention calculation on the image block feature sequences based on the prompt query vectors to update the text prompt features; performing image-to-text cross-attention calculation on the image block feature sequences based on the key vectors and the value vectors of the updated text prompt features to update the image block feature sequences; updating the text prompt features and the updated image block feature sequences for a preset number of times to obtain fused image features and fused text prompt features. 4.The method of claim 1, wherein, The method comprises the following steps: performing up-sampling processing on the fused image features to obtain high-resolution image feature mappings matching the original image spatial resolution; mapping the preset output token into a dynamic linear classifier through a multi-layer perception machine; inputting the high-resolution image feature mappings into the dynamic linear classifier to calculate mask foreground probabilities of each spatial position in the image, and generating an image mask based on the foreground probabilities. 5.The method of claim 1, wherein, The method comprises the following steps: The fused image feature is multiplied with the image mask to obtain a multiplication result; The multiplication result is averaged to obtain an image global feature. 6.The method of claim 1, wherein, The text global feature is extracted from the text token feature sequence, including: Any one of the following features is selected from the last layer of the output text token feature sequence as the text global feature, including: an embedding vector of a first token in the text token feature sequence, an embedding vector of a token used to identify the end of the text in the text token feature sequence, and a vector obtained by averaging all embedding vectors of tokens in the text token feature sequence. 7.The method of claim 1, wherein, The contrast loss is a bidirectional InfoNCE loss, which includes an image-to-text contrast loss and a text-to-image contrast loss, the mask loss is a Dice loss, and the joint loss function is a weighted sum of the contrast loss and the mask loss.

8. The device for training a text-image alignment model, characterized in that, The method comprises: An encoding unit is configured to encode input image data and text data respectively to obtain an image block feature sequence and a text token feature sequence, wherein a label of the image data includes a text description of the image and a label mask related to the description; A bidirectional fusion unit is configured to perform bidirectional feature interaction between the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features; An image mask generation unit is configured to input the fused image features, the fused text prompt features, and a preset output token into a SAM mask decoder to generate an image mask; A filtering unit is configured to filter the fused image features based on the image mask to extract image global features related to the text description; An extraction unit is configured to extract a text global feature from the text token feature sequence; A construction unit is configured to calculate a contrast loss based on the image global feature and the text global feature, and construct a joint loss function in combination with a mask loss of the image mask and the label mask; A training unit is configured to perform end-to-end training on the model based on the joint loss function until the model converges.

9. A computer device, comprising: The memory, the processor, and a computer program stored in the memory and executable on the processor are included, and the processor implements the training method of the image-text alignment model according to any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions, which, when executed by a processor, cause the processor to execute the training method of the image-text alignment model according to any one of claims 1-7.

Images