Text matching model training method, device, storage medium and program product

Abstract

The application discloses a kind of training method, equipment, storage medium and program product of graphic-text matching model, to facilitate the matching accuracy of graphic-text matching model.In the present application, a plurality of sample pairs are obtained, and a first text encoder of a graphic-text matching model is used to process a first description text, and a second text encoder of the graphic-text matching model is used to process a second description text to obtain corresponding text features;Image features of sample images are extracted using an image encoder of the graphic-text matching model;The graphic-text matching model is trained based on the text features and the image features.In the present application, different position encoding strategies are used for long and short texts, respectively, wherein the short text directly reuses the original position encoding of pre-training, and the long text expands the position encoding by segmentation interpolation.This process improves the alignment capability of the model in long text tasks, while retaining the strong generalization capability of short text tasks, improving the matching accuracy of the graphic-text matching model.

Description

Technical Field

[0001] This invention relates to the field of image-text matching, and more particularly to a training method, device, storage medium, and program product for an image-text matching model. Background Technology

[0002] With the rapid development of multimodal artificial intelligence technology, image-text matching models (such as Contrastive Language-Image Pre-training, CLIP) have achieved significant performance improvements in image-text alignment tasks. However, existing technologies still have significant performance bottlenecks when handling long text and image matching tasks, especially in feature extraction and alignment of long texts.

[0003] Existing image-text matching models are typically based on pre-trained text encoders and image encoders, aligning image and text features through contrastive learning. These models perform well on short text tasks, but on long text tasks, due to the limitation of text input length (usually less than 77 tokens, where a token is the smallest semantic unit in the text), the models cannot fully capture the detailed information in long texts, resulting in poor alignment and thus affecting the matching accuracy of the image-text matching model. Summary of the Invention

[0004] The purpose of this invention is to provide a training method, device, storage medium, and program product for a text-image matching model, in order to solve the technical problems in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for training a text-image matching model, the method comprising:

[0006] Acquire multiple sample pairs; each sample pair includes a sample image and corresponding sample input data; the sample input data corresponding to any sample image includes sample description text describing the image content contained in the sample image;

[0007] The first descriptive text is processed using a first text encoder of the image-text matching model, and the second descriptive text is processed using a second text encoder of the image-text matching model to obtain the corresponding text features. The positional encoding in the first text encoder reuses the original positional encoding of the pre-trained text encoder. The positional encoding of the first n positions in the second text encoder reuses the first n positions in the original positional encoding. The positional encoding of the (n+1)th to mth positions in the second text encoder is obtained by interpolating the encoding of the (n+1)th position and beyond in the original positional encoding. n and m are preset values. The first descriptive text is a sample descriptive text with a token count less than or equal to the first preset value. The second descriptive text is a sample descriptive text with a token count greater than the first threshold.

[0008] Image features of sample images are extracted using an image encoder based on an image-text matching model;

[0009] The image-text matching model is trained based on text features and image features.

[0010] In this embodiment, different positional encoding strategies are employed for short and long texts. Short texts directly reuse the pre-trained original positional encoding, ensuring the alignment effect remains unaffected. Long texts, however, have their positional encoding expanded through segmented interpolation. This preserves local semantic relevance while addressing the limitation of long text input length, enabling more accurate capture of detailed information within the text. This process enhances the model's alignment capability in long text tasks while retaining its strong generalization ability in short text tasks, thus improving the matching accuracy of the image-text matching model.

[0011] In some possible embodiments, an image encoder using a text-image matching model extracts image features from the sample image, including:

[0012] For each transformer block in the image encoder, obtain the region cueing parameters corresponding to the transformer block;

[0013] Obtain the initial input sequence for the transformer block;

[0014] The region hint parameters are inserted into the initial input sequence to obtain the final input sequence of the transformer block;

[0015] The final input sequence is fed into the transformer block, which processes the final input sequence to obtain the image features of the sample image.

[0016] In this embodiment, by introducing region cueing parameters into each Transformer block of the image encoder, the model's attention to local details of the image can be enhanced. At the same time, by interpolating the image sequence, it is ensured that the feature information between different depth layers does not interfere with each other, thereby avoiding the feature coupling problem, improving the detail capture capability of image feature extraction and the consistency of multi-layer features, and solving the problem of insufficient local information extraction in the prior art.

[0017] In some possible embodiments, the transformer block processes the final input sequence to obtain image features of the sample image, including:

[0018] Obtain the one-way attention mask corresponding to the cue parameters for each region;

[0019] Each unidirectional attention mask is used to mask the image encoding in the final input sequence to obtain the image encoding of each region.

[0020] The image encoding of each region is processed based on a multi-head self-attention mechanism to obtain the image features of each region.

[0021] In this embodiment, a unidirectional attention mask is introduced into the Transformer block to limit the scope of the self-attention mechanism, ensuring that each region cue only focuses on a specific image region, thereby avoiding interference or coupling of feature information between different depth layers. Simultaneously, the multi-head self-attention mechanism can capture features at different scales in the image sequence, enhancing the model's ability to capture global and local information, and improving the accuracy of image feature extraction and model performance. This process enables the model to extract features more accurately when processing complex images and achieve higher alignment accuracy and generalization ability in image-text matching tasks, effectively alleviating the feature alignment bias problem in existing technologies.

[0022] In some possible embodiments, the image-text matching model is trained based on text features and image features, including:

[0023] Calculate the contrast loss between text features and image features, and then calculate the gradient based on the contrast loss;

[0024] The model parameters of the image-text matching model are updated based on the gradient; the model parameters include the region hint parameters corresponding to the transformer block.

[0025] In this embodiment, the contrast loss between text features and image features is calculated, and the gradient is calculated based on the loss. The model parameters, including the region hint parameter, in the image-text matching model are then updated, enabling the model to learn more accurate feature representations. At the same time, the model can capture local details more accurately when processing images, thereby achieving higher alignment accuracy in the image-text matching task.

[0026] In some possible embodiments, the image-text matching model is trained based on text features and image features, including:

[0027] The transformer blocks in the first text encoder, the second text encoder, and the image encoder are grouped for processing; each group contains the same number of transformer blocks.

[0028] Based on the grouping results, the text features and image features are respectively subjected to group feature aggregation, linear transformation and layer normalization to obtain the corresponding text group feature matrix and image group feature matrix.

[0029] Calculate the group contrast loss between the feature matrix of each text group and the feature matrix of each image group;

[0030] The loss of the image-text matching model is determined based on group contrast loss, and the training of the image-text matching model is considered complete when the loss converges.

[0031] In this embodiment, by grouping the Transformer blocks in the first text encoder, second text encoder, and image encoder, and then performing group feature aggregation, linear transformation, and layer normalization on the text features and image features based on the grouping results, the accuracy and consistency of feature representation can be effectively improved. By calculating the group contrast loss between each group of text feature matrices and the image feature matrices, and determining the total loss of the image-text matching model based on these losses, the model can progressively optimize the feature alignment effect at multiple depth levels. This not only improves the alignment accuracy of long text and image features but also enhances the training efficiency and generalization ability of the model through the grouping optimization strategy.

[0032] In some possible embodiments, the loss of the image-text matching model is determined based on group contrast loss, including:

[0033] Obtain the group loss weight corresponding to each group contrast loss, and determine the total contrast loss by summing the products of each group contrast loss and its corresponding group loss weight;

[0034] Obtain the first text code corresponding to the first descriptive text, the second text code corresponding to the second descriptive text, and the image code corresponding to the sample image;

[0035] Calculate the first L2 loss between the first text encoding and the image encoding, and calculate the second L2 loss between the second text encoding and the image encoding;

[0036] The sum of the first L2 loss, the second L2 loss, and the total contrast loss is determined as the loss of the image-text matching model.

[0037] In this embodiment, by assigning weights to each set of contrastive losses and calculating the total contrastive loss, the model can balance the contributions of features at different levels and improve the accuracy of feature representation. Simultaneously, by calculating the L2 loss between the first text encoding, the second text encoding, and the image encoding, the model can further optimize the alignment of text and image features, reducing bias in feature representation. Finally, by combining the total contrastive loss with the L2 loss, the model can comprehensively optimize feature representation at multiple levels, improving the accuracy and generalization ability of the image-text matching task.

[0038] In some possible embodiments, multiple sample pairs are obtained, including:

[0039] Obtain a general dataset; the general dataset contains multiple image-text pairs;

[0040] The images in the general dataset are re-labeled using a large language model to obtain corresponding image description text; the number of tokens in the image description text is greater than a second preset value;

[0041] The relabeled general dataset was processed to obtain multiple sample pairs.

[0042] In this embodiment, by re-labeling image description text using a large language model, more detailed and higher-quality descriptions can be generated, overcoming the limitations of short text descriptions and improving the expressive power of text features. Simultaneously, data filtering further ensures the purity and relevance of the dataset, removing low-quality or irrelevant image-text pairs, thereby improving the quality of the training data.

[0043] In some possible embodiments, after obtaining multiple sample pairs, the method further includes:

[0044] The sample pairs are divided into multiple datasets of different sizes.

[0045] In this embodiment, by dividing the sample pairs into datasets of different sizes, different training strategies can be adopted for datasets of different sizes, thereby improving training efficiency. Simultaneously, datasets of different sizes can be used for different training stages, helping the model to gradually optimize at different levels. Small datasets allow for rapid adjustment of model parameters, while large datasets can further enhance the model's generalization ability.

[0046] Secondly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods provided in the first aspect embodiment of this application.

[0047] Thirdly, another embodiment of this application provides a computer-readable storage medium storing a computer program for causing a computer to perform any of the methods provided in the first aspect of the embodiments of this application.

[0048] Fourthly, another embodiment of this application also provides a computer program product, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform any of the methods provided in the first aspect embodiment above.

[0049] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a CLIP model framework in the prior art;

[0051] Figure 2 This is a schematic diagram illustrating the overall process of a training method for a text-image matching model provided in an embodiment of this application.

[0052] Figure 3 for Figure 2 A flowchart illustrating a practical manifestation of step 103 in the training method of the provided image-text matching model;

[0053] Figure 4 This is a schematic diagram illustrating a masking process for image encoding in the final input sequence using a one-way attention mask, provided as an embodiment of this application.

[0054] Figure 5 for Figure 2 A flowchart illustrating a practical manifestation of step 104 in the training method of the provided image-text matching model;

[0055] Figure 6 for Figure 2 A flowchart illustrating another practical manifestation of step 104 in the training method of the provided image-text matching model.

[0056] Figure 7 This is a schematic diagram of the architecture of a text-image matching model provided in an embodiment of this application;

[0057] Figure 8 for Figure 2 A flowchart illustrating an actual manifestation of step 101 in the training method of the provided image-text matching model;

[0058] Figure 9 This is a schematic diagram illustrating the effect of a text-to-image task provided in an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of an electronic device for a training method of a text-image matching model provided in an embodiment of this application. Detailed Implementation

[0060] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0061] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0062] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0063] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0065] Please refer to Figure 1 This is a schematic diagram of a CLIP model framework in the prior art. Figure 1 As shown, CLIP is a multimodal alignment method. Its core idea is to pre-train using a large amount of paired image and text data to learn the alignment relationships between images and text. The CLIP model contains two modalities, an image encoder and a text encoder, which transform image and text inputs into low-dimensional vector representations. The CLIP model generates predictions by calculating the cosine similarity between text-image pairs, thus adapting to many zero-shot tasks, such as image-text retrieval and image-text generation.

[0066] While CLIP performs well in text-image alignment tasks, its limitation by short text inputs (typically less than 77 tokens) prevents it from extracting sufficient detail from the text. This not only impacts other potential applications of text encoders but also further restricts the detail extraction capabilities of image encoders. Due to these shortcomings, text-image alignment can exhibit significant biases, causing both image and text encoders to focus on the overall concept while ignoring local details, thus affecting the accuracy of text-image matching.

[0067] To address the aforementioned problems, this application provides a training method, device, storage medium, and program product for an image-text matching model, which solves the problems described above. The inventive concept of this application can be summarized as follows: acquiring multiple sample pairs; processing a first descriptive text using a first text encoder of the image-text matching model, and processing a second descriptive text using a second text encoder of the image-text matching model to obtain corresponding text features; extracting image features of the sample images using an image encoder of the image-text matching model; and training the image-text matching model based on the text features and image features.

[0068] For ease of understanding, the training method of a text-image matching model provided in this application embodiment will be described in detail below with reference to the accompanying drawings:

[0069] like Figure 2 The diagram shown is a schematic representation of the overall process of training a text-image matching model according to an embodiment of this application, wherein:

[0070] In step 101: Obtain multiple sample pairs.

[0071] Each sample pair includes a sample image and corresponding sample input data; the sample input data corresponding to any sample image includes sample description text that describes the image content contained in the sample image.

[0072] In this embodiment, sample pairs are the basic data units used to train the image-text matching model. Each sample pair consists of two parts: a sample image and corresponding sample input data. The sample image can be any type of image (such as a photograph, illustration, chart, etc.), while the sample input data includes descriptive text related to the image content. This descriptive text provides a detailed description of the image content, such as information about objects, scenes, colors, and actions contained in the image. The construction of sample pairs aims to provide the model with the correlation between images and text, enabling it to learn how to effectively match image features with text features. By using sample pairs, the model can learn the complex relationship between images and text during training and gradually optimize its alignment ability, ultimately achieving efficient image-text matching in practical applications.

[0073] The constructed sample pairs can be used not only for model training but also for model validation and testing. In the validation phase, the sample pairs are used to evaluate the model's alignment performance, ensuring it achieves the expected results in both long and short text tasks. In the testing phase, the sample pairs are used to verify the model's generalization ability in real-world applications, ensuring it can handle image-text matching tasks in various complex scenarios.

[0074] In some possible implementations, to obtain high-quality sample pairs and improve the model's ability to process long texts, sample images and corresponding sample descriptive texts can be extracted from existing datasets based on preset extraction rules. For example, image-text pairs with more than a preset value of descriptive text tokens can be extracted from general long text image datasets (such as SA1B-Dense-Caption, LAION-5B, LAION-400M, etc.) as sample pairs.

[0075] In step 102: the first descriptive text is processed using the first text encoder of the image-text matching model, and the second descriptive text is processed using the second text encoder of the image-text matching model to obtain the corresponding text features.

[0076] In this text encoder, the positional encoding reuses the original positional encoding of the pre-trained text encoder; in the second text encoder, the positional encoding of the first n bits reuses the first n bits of the original positional encoding, and the positional encoding of the (n+1)th to mth bits in the second text encoder is obtained by interpolating the encoding of the (n+1)th bit and beyond in the original positional encoding; n and m are preset values; the first description text is the sample description text with a token count less than or equal to the first preset value; the second description text is the sample description text with a token count greater than the first threshold.

[0077] In this embodiment, the text encoder is a core component of the image-text matching model, used to convert the input text into feature vectors that the model can understand. These feature vectors can capture the semantic information of the text, thereby achieving accurate matching between text and images. This embodiment uses two different text encoders: a first text encoder and a second text encoder. The first text encoder is used to process short text, while the second text encoder is used to process long text.

[0078] In the first text encoder, the positional encoding directly reuses the pre-trained original positional encoding, ensuring the alignment of short texts. In the second text encoder, the positional encoding is divided into two parts: the first n positions reuse the first n positions from the original positional encoding, while the (n+1)th to mth positions are obtained by interpolating the (n+1)th position and beyond from the original positional encoding. This interpolation method effectively extends the length of the positional encoding, adapting it to the needs of long text inputs. In this way, the second text encoder can retain the alignment capability of short texts while processing long texts, while simultaneously improving the feature extraction capability of long texts.

[0079] For example, in some possible embodiments, the raw positional encoding of the pre-trained text encoder can be the positional encoding corresponding to the pre-trained CLIP model, where n can be 20, m can be 268, and the first preset value can be 77. The positional encoding of the second text encoder can be calculated according to the following formula:

[0080]

[0081] Where PE is the original pre-trained positional encoding (e.g., [L, d]), with dimensions [L, d], where L is the maximum sequence length supported by pre-training. Concat is the concatenation operation, joining the original encodings of the first 20 positions with the subsequently interpolated encodings. Interpol is the interpolation function used to extend the positional encodings to longer sequences. q is the interpolation extension factor, here q = 4, indicating that each original position is extended to 4 interpolation positions. i is the index of the new interpolated position (relative to the extended sequence). Rounding down i / q determines the index of the interpolation reference point in the original positional encoding. λ is the interpolation weight coefficient.

[0082] The purpose of this application is to address the imbalance in processing capabilities between short and long texts in existing technologies. By designing different positional encoding strategies, different encoding methods are employed for short and long texts respectively. Short texts directly reuse the pre-trained original positional encoding, ensuring that the alignment effect of short texts is not affected. Long texts, on the other hand, have their positional encoding expanded through segmented interpolation. The first n positional encodings retain the local semantic relationships (such as word order and phrase structure) captured during text encoder training, avoiding a decrease in short text comprehension ability due to reinitialization. Simultaneously, this solves the problem of limited input length for long texts, enabling more accurate capture of detailed information within long texts.

[0083] In step 103: the image encoder of the image-text matching model is used to extract the image features of the sample image.

[0084] In this embodiment, the image encoder is a key component of the image-text matching model, used to convert the input image into feature vectors that the model can understand. These feature vectors can capture key visual information of the image, such as color, shape, texture, and object position, thereby achieving accurate matching between the image and text. The image encoder can automatically learn the feature representation of the image through deep learning techniques (such as convolutional neural networks, CNNs, or transformer models), enabling it to be compared and matched with text features in the same feature space, thus achieving the image-text alignment task.

[0085] In some possible embodiments, the image encoder can be implemented based on a pre-trained deep learning model (such as VisionTransformer, ViT). Specific steps may include:

[0086] Image preprocessing: Standardize the sample images, such as adjusting the image size and normalizing pixel values, to adapt them to the model input requirements.

[0087] Feature extraction: The preprocessed image is input into an image encoder, which extracts feature vectors from the image through network structures such as convolutional layers and Transformer layers. These feature vectors typically include global features (such as the overall layout of the image) and local features (such as object details in the image).

[0088] In step 104: The image-text matching model is trained based on text features and image features.

[0089] In this embodiment, the image-text matching model is trained through contrastive learning, mapping image features and text features to the same feature space to achieve accurate image-text matching. By optimizing model parameters during training, matched image-text pairs are made closer in the feature space, while mismatched pairs are made further apart, thereby improving the model's performance in image-text matching tasks. Optimizing the alignment of long text features with image features enhances the model's performance in long text tasks while ensuring that performance in short text tasks remains unaffected, maintaining the model's strong generalization ability in short text tasks.

[0090] In some possible implementations, to further improve training performance, data augmentation techniques (such as image rotation, cropping, and text synonym replacement) can be used to increase the diversity of training data and improve the model's generalization ability. Alternatively, hyperparameters such as learning rate and batch size can be adjusted to optimize the convergence speed and final performance of the training process. Furthermore, short texts and images can be pre-trained first, followed by incremental training on long texts and images to gradually improve the model's long text processing capabilities.

[0091] Based on the above technical solution, this application employs different positional encoding strategies for short and long texts. Short texts directly reuse the pre-trained original positional encoding, ensuring that the alignment effect of short texts is not affected. Long texts, on the other hand, have their positional encoding expanded through segmented interpolation. This preserves local semantic relevance while addressing the limitation of long text input length, enabling more accurate capture of detailed information within long texts. This process improves the model's alignment capability in long text tasks while retaining its strong generalization ability in short text tasks, thus enhancing the matching accuracy of the image-text matching model.

[0092] Please refer to Figure 3 ,for Figure 2The provided diagram illustrates a practical implementation of step 103 in the training method for an image-text matching model. Based on the above embodiments, in some possible embodiments, step 103, which involves extracting image features from the sample image using the image encoder of the image-text matching model, includes:

[0093] In step 201: For each transformer block in the image encoder, obtain the region cue parameters corresponding to the transformer block.

[0094] In this embodiment, the region cueing parameter is a learnable parameter in the image encoder, used to enhance the model's ability to focus on local details of the image. Since the self-attention mechanism in the traditional Transformer architecture primarily focuses on global features and has limited ability to capture local details, this embodiment, by introducing the region cueing parameter, helps the model better focus on key regions in the image, thereby improving the accuracy of feature extraction. This addresses the problem of insufficient local information extraction in existing technologies and enhances the model's ability to capture image details.

[0095] In some possible embodiments, the region hinting parameter is a configuration item used to dynamically adjust the output behavior of a model or system within a specific data region, and its specific uses may include:

[0096] 1) Divide the data or operation space into different regions (such as geographical range, numerical range, etc.) by preset rules, and set parameter thresholds or response strategies independently for each region.

[0097] 2) Dynamic response and interactive optimization: Adjust parameter values ​​based on real-time input or context to adapt to the feature requirements of different regions and improve the accuracy and adaptability of model output.

[0098] In this way, the model can capture local features more accurately when processing complex images, thereby achieving higher alignment accuracy in image-text matching tasks.

[0099] In step 202: Obtain the initial input sequence for the transformer block.

[0100] In this embodiment, the initial input sequence serves as the starting input for the Transformer block to process image data, containing basic feature information of the image. In the image encoder, the initial input sequence is generated by dividing the image into multiple patches and then performing a linear projection on each patch. This representation transforms the two-dimensional structure of the image into a one-dimensional sequence for the Transformer model to process. The initial input sequence provides the model with a structured input, enabling it to capture global and local features in the image through a self-attention mechanism.

[0101] In some possible embodiments, the process of obtaining the initial input sequence may include:

[0102] Image segmentation: Dividing an input image into multiple fixed-size patches, such as 16x16 pixel patches.

[0103] Linear projection: Perform a linear projection on each small block, transforming it into a vector of fixed dimensions.

[0104] Add position encoding: In order to preserve the two-dimensional structural information of the image, position encoding is added to the vector of each small block.

[0105] Serialization: Arrange the vectors of all small blocks in order to form a one-dimensional sequence, which serves as the initial input sequence for the Transformer block.

[0106] In step 203: the region hint parameters are inserted into the initial input sequence to obtain the final input sequence of the transformer block.

[0107] For example, in the l-th transformer block of the image encoder, the M region cue parameters corresponding to the l-th transformer block can be inserted into the initial input sequence ([CLS], P1, ..., P...). N In the definition of the l-th layer transformer block, the final input sequence is used. Where P i (i∈[1,N]) represents the encoding of the i-th image. This indicates the hint for the j-th region.

[0108] In this embodiment, by inserting the region hint parameter into the input sequence of the transformer block, the region hint parameter can be combined with the image sequence, thereby highlighting the features of specific regions during the feature extraction process, enhancing the model's ability to focus on local details of the image, solving the problem of insufficient local information extraction in the prior art, and improving the model's ability to capture image details.

[0109] In step 204: the final input sequence is input into the transformer block, which processes the final input sequence to obtain the image features of the sample image.

[0110] For example, when the final input sequence of the l-th transformer block is At that time, after processing by the transformer block, the image features of the sample image are obtained. Among them, X l The image features output by the l-th transformer block, Let D be the set of real numbers and D be the channel dimension.

[0111] In this embodiment, the Transformer block is a deep learning architecture based on a self-attention mechanism for processing sequential data. The Transformer block in the image encoder captures global and local features in the image sequence through a self-attention mechanism. Each Transformer block consists of multi-head self-attention and a feed-forward network, enabling progressive optimization of feature representation. The self-attention mechanism captures the relationships between any positions in the sequence, thereby extracting global features. Simultaneously, through multi-head self-attention, the model can focus on local features at different scales.

[0112] In some possible embodiments, after the l-th layer transformer block outputs image features, the region cue parameters can be replaced from the l-th layer parameters to the (l+1)-th layer region cue parameters according to a formula to determine the final input sequence of the (l+1)-th layer transformer block:

[0113]

[0114] Based on the above technical solution, this application embodiment introduces region cueing parameters in each Transformer block of the image encoder, which can enhance the model's attention to local details of the image; at the same time, by interpolating the image sequence, it ensures that the feature information between different depth layers does not interfere with each other, thereby avoiding the feature coupling problem, improving the detail capture capability of image feature extraction and the consistency of multi-layer features, and solving the problem of insufficient local information extraction in the prior art.

[0115] In some possible embodiments, the process mentioned in step 204, in which the transformer block processes the final input sequence to obtain the image features of the sample image, may specifically include the following steps:

[0116] In step 301: Obtain the one-way attention mask corresponding to the cue parameters of each region.

[0117] In this embodiment, a one-way attention mask is a masking mechanism used to limit the computational scope of the self-attention mechanism in a Transformer block. It sets the attention weights at specific locations to negative infinity, ignoring these locations during attention computation, thus ensuring the model only focuses on relevant information. The one-way attention mask limits the scope of the image encoder's self-attention mechanism, ensuring that each region cue only focuses on specific image regions. This effectively avoids interference or coupling of feature information between different depth layers, thereby improving the model's ability to capture local details. Simultaneously, the one-way attention mask can also help the model extract features more efficiently when processing complex images, improving the accuracy of feature representation.

[0118] In step 302: each one-way attention mask is used to mask the image encoding in the final input sequence to obtain the image encoding of each region.

[0119] For example, please refer to Figure 4 This is a schematic diagram illustrating a method for masking the image encoding in the final input sequence using a one-way attention mask, as provided in an embodiment of this application.

[0120] In this embodiment, the one-way attention mask allows region cueing parameters to focus on specific region codes while maintaining the integrity of the original region codes. For example... Figure 4 As shown, each row's query vector Q corresponds to a mask vector Mask, defining the key-value positions (columns) that can be followed, where:

[0121] The [CLS] token allows you to follow yourself and all area prompts (R1~R). m ) and all area codes (P1~P nIts function is to achieve global information aggregation and coordinate semantic relationships between regions;

[0122] Area hint R j Allow attention to the region-coded block (P_b) that is itself and its associated regions. j ~P_b j +k), its masking function is to force R j Focus only on the specified local area to avoid interference from irrelevant areas;

[0123] Region code P i Allows attention to all non-regional prompt locations (i.e., other Ps) i And [CLS]), its function is to maintain the independence of the original encoding and prevent it from being overwritten by the region hint.

[0124] The unidirectional attention mask vector is defined as follows:

[0125]

[0126] Each region prompts R j Responsible for encoding k consecutive regions, where (Round down).

[0127] In this embodiment, by designing a specific unidirectional attention mask for each image region, the model can focus on relevant contextual information for different regions when processing the input sequence, thereby obtaining more region-specific location encodings. By limiting the scope of the self-attention mechanism, the unidirectional attention mask can enhance the ability to capture local image features.

[0128] In step 303: The image encoding of each region is processed based on the multi-head self-attention mechanism to obtain the image features of each region.

[0129] In this embodiment, after obtaining the region code for each region, multiple independent attention heads are used to compute the dynamic associations between different regions in parallel. Each attention head limits the interaction range through a unidirectional attention mask (e.g., only allowing the current region to focus on specific neighboring regions or global semantic nodes), extracts features of different dimensions (e.g., color, texture, shape), and finally concatenates and fuses the outputs of the multiple heads to obtain region features that can both preserve local details (e.g., edge sharpness) and integrate global context (e.g., positional relationships between objects). For example, when processing a face image, one attention head may focus on the eye contour through mask constraints, while another head associates the light and dark transitions between the bridge of the nose and the cheeks, and the combined result forms a highly discriminative feature representation.

[0130] This application's embodiments limit the scope of the self-attention mechanism by introducing a unidirectional attention mask in the Transformer block, ensuring that each region cue only focuses on a specific image region, thereby avoiding interference or coupling of feature information between different depth layers. Simultaneously, the multi-head self-attention mechanism can capture features at different scales in the image sequence, enhancing the model's ability to capture global and local information, and improving the accuracy of image feature extraction and model performance. This process enables the model to extract features more accurately when processing complex images and achieve higher alignment accuracy and generalization ability in image-text matching tasks, effectively alleviating the feature alignment bias problem in existing technologies.

[0131] Please refer to Figure 5 ,for Figure 2 The provided flowchart illustrates one practical manifestation of step 104 in the training method for an image-text matching model. Based on the above embodiments, in some possible embodiments, step 104, which involves training the image-text matching model based on text features and image features, may specifically include the following steps:

[0132] In step 401, the contrast loss between text features and image features is calculated, and the gradient is calculated based on the contrast loss.

[0133] In this embodiment, contrastive loss is a loss function used to measure the similarity between text features and image features. It optimizes the model's feature representation by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. During training, by calculating the contrastive loss between text features and image features and calculating the gradient based on this loss, the alignment accuracy and generalization ability of the model can be effectively improved.

[0134] In step 402, the model parameters of the image-text matching model are updated according to the gradient; the model parameters include the region hint parameters corresponding to the transformer block.

[0135] In this embodiment, gradient update is a crucial step in machine learning for optimizing model parameters. By calculating the partial derivatives (gradients) of the loss function with respect to the model parameters and adjusting the parameters according to certain update rules, the loss function is minimized. Gradient update effectively optimizes model parameters and improves model performance. In information matching models, gradient update can optimize the alignment between text features and image features, enabling the model to achieve higher alignment accuracy and generalization ability in image-text matching tasks. Simultaneously, by updating the region cue parameters in the Transformer block, the model can more accurately capture local details in the image.

[0136] This application embodiment calculates the contrast loss between text features and image features, and calculates the gradient based on the loss, thereby updating the model parameters in the image-text matching model, including the region hint parameter. This enables the model to learn more accurate feature representations, and at the same time, the model can more accurately capture local details when processing images, thus achieving higher alignment accuracy in image-text matching tasks.

[0137] Please refer to Figure 6 ,for Figure 2 The flowchart illustrates another practical manifestation of step 104 in the provided training method for an image-text matching model. Based on the above embodiments, in some possible embodiments, step 104, which involves training the image-text matching model based on text features and image features, may specifically include the following steps:

[0138] In step 501, the transformer blocks in the first text encoder, the second text encoder, and the image encoder are grouped for processing.

[0139] For example, assuming that the first text encoder, the second text encoder, and the image encoder are all 24-layer structures, and each layer has only one transformer block, then the transformer blocks of the four adjacent layers can be divided into a group, for a total of six groups. The first group contains the transformer blocks of layers 1-4, and the last group contains the transformer blocks of layers 20-24.

[0140] In this embodiment, grouping is a strategy that divides Transformer blocks into several groups according to certain rules, with each group containing the same number of Transformer blocks. The purpose is to optimize computational efficiency and the model's expressive power. Through grouping, the model can progressively optimize feature representations at multiple depth levels, enhancing its ability to capture global and local information and reducing computational complexity. Simultaneously, grouping allows the model to perform parallel computation at multiple levels, improving training and inference speed.

[0141] In step 502, based on the grouping results, the text features and image features are subjected to group feature aggregation, linear transformation and layer normalization to obtain the corresponding text group feature matrix and image group feature matrix.

[0142] In this embodiment, group feature aggregation, linear transformation, and layer normalization are methods for optimizing text and image features. After grouping the transformer blocks, the text and image feature matrices output by each transformer block are aggregated separately, then a linear transformation is performed to adjust the feature dimensions, and finally layer normalization is used to standardize the features, thereby improving the model's performance and generalization ability.

[0143] Among these processes, group feature aggregation captures feature information at different levels, linear transformation adjusts feature dimensions to adapt to subsequent processing, and layer normalization standardizes features, reduces internal covariate shifts, and improves model training efficiency and performance. Through these processing steps, the model can more accurately align text and image features, thereby achieving higher accuracy in text-image matching tasks.

[0144] In step 503, the group contrast loss between the feature matrix of each text group and the feature matrix of the image group is calculated.

[0145] In this embodiment, group contrast loss is a loss function used to measure the similarity between the text group feature matrix and the image group feature matrix. It optimizes the model's feature representation by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. In multimodal tasks, group contrast loss can effectively improve the alignment accuracy of text and image features, thereby achieving higher performance and generalization ability in image-text matching tasks.

[0146] Due to the high complexity of the feature space of long texts, existing technologies only establish correlations on the visual language features of the last layer, leading to asymmetry in the feature information of image and text features at the output layer. Therefore, this application introduces a grouped feature alignment strategy, which not only aligns the features at the last layer but also progressively optimizes the alignment effect of image and text features at multiple depth levels. This ensures the consistency of image and text features across multiple depth levels, avoids the problem of feature information asymmetry, improves the performance of long text tasks, and enhances the model's generalization ability and detail extraction capability in complex scenarios.

[0147] In step 504, the loss of the image-text matching model is determined based on the group contrast loss, and the image-text matching model training is completed when the loss converges.

[0148] In this embodiment, after determining the group contrast loss, the loss of the image-text matching model is determined based on the group contrast loss. The change in the loss of the image-text matching model is key to determining whether the model has converged. When the loss value stabilizes and reaches the preset convergence condition, the model can be considered to have been successfully trained.

[0149] This application embodiment groups the Transformer blocks in the first text encoder, second text encoder, and image encoder, and performs group feature aggregation, linear transformation, and layer normalization on the text features and image features based on the grouping results, which effectively improves the accuracy and consistency of feature representation. By calculating the group contrast loss between the feature matrices of each text group and the feature matrices of the image group, and determining the total loss of the image-text matching model based on these losses, the model can progressively optimize the feature alignment effect at multiple depth levels, which not only improves the alignment accuracy of long text and image features, but also improves the training efficiency and generalization ability of the model.

[0150] In some possible embodiments, the step 504, which involves determining the loss of the image-text matching model based on group contrast loss, may specifically include the following steps:

[0151] In step 601, the group loss weight corresponding to each group contrast loss is obtained, and the sum of the products of each group contrast loss and the corresponding group loss weight is determined as the total contrast loss.

[0152] In this embodiment, the group loss weights are used to balance the contributions of different groups, enabling the model to focus more on important feature groups during training. By obtaining the group loss weights corresponding to each group's contrastive loss and determining the sum of the products of each group's contrastive loss and its corresponding group loss weight as the total contrastive loss, it can be ensured that the model does not ignore the feature representations of certain key groups during optimization.

[0153] The introduction of group loss weights allows the model to perform differentiated optimization based on the importance of each group. Different groups may have different impacts on the model's final performance. By assigning appropriate weights to each group, the model can ensure that it pays more attention to those groups that have a greater impact on performance during the optimization process. This not only improves the overall performance of the model but also enhances its stability and adaptability when dealing with complex data.

[0154] In step 602, the first text code corresponding to the first descriptive text, the second text code corresponding to the second descriptive text, and the image code corresponding to the sample image are obtained.

[0155] In the embodiments of this application, when extracting features using a first text encoder, a second text encoder, or an image encoder, it is first necessary to use the corresponding position encoder in each encoder to convert the text and image into the corresponding first text code, second text code, and image code. These codes can capture the core semantic information of the text and image, providing a foundation for subsequent feature extraction.

[0156] In step 603, the first L2 loss between the first text encoding and the image encoding is calculated, and the second L2 loss between the second text encoding and the image encoding is calculated.

[0157] In this embodiment, by calculating L2 loss, the model can optimize the alignment of text and image features. The first L2 loss focuses on matching short text with images, while the second L2 loss focuses on matching long text with images. This differentiated evaluation allows the model to achieve efficient feature alignment and optimization when processing text of different lengths. The L2 loss evaluates the model's performance by calculating the average of the squared differences between the predicted and true values. In the image-text matching model, this calculation can capture the differences between text and image features, helping the model learn more accurate feature representations.

[0158] In step 604, the sum of the first L2 loss, the second L2 loss, and the total contrast loss is determined as the loss of the image-text matching model.

[0159] For example, the loss of the image-text matching model can be calculated using the following formula:

[0160]

[0161] Where L is the loss of the image-text matching model. For the total comparison loss, ω i Let L be the group loss weight for the i-th group. mi For comparison of group losses in group i, L short For the first L2 loss, L long This is the second L2 loss.

[0162] In this embodiment, the comprehensive loss is formed by summing the first L2 loss, the second L2 loss, and the total contrastive loss to create a comprehensive loss function. The first L2 loss measures the difference between short text encoding and image encoding, the second L2 loss measures the difference between long text encoding and image encoding, and the total contrastive loss optimizes the feature representation through grouped contrastive learning. In this way, the model can optimize in multiple aspects simultaneously, ensuring accurate alignment of short text, long text, and image features.

[0163] The purpose of the comprehensive loss is to balance the contributions of different loss terms, enabling the model to achieve efficient feature alignment and optimization when processing text descriptions of varying lengths. By appropriately allocating weights, the model can simultaneously focus on the matching effects between short texts, long texts, and images, improving overall performance. This approach not only improves the model's accuracy but also enhances its stability and adaptability in complex scenarios.

[0164] In practice, the first L2 loss, the second L2 loss, and the total contrastive loss are calculated separately. Then, based on task requirements and experience, they are summed to obtain the comprehensive loss. Finally, backpropagation is used to optimize the comprehensive loss and update the model parameters, ensuring that the model's performance gradually improves during training and ultimately reaches its optimal state.

[0165] In this embodiment, by assigning weights to each set of contrastive losses and calculating the total contrastive loss, the model can balance the contributions of features at different levels and improve the accuracy of feature representation. Simultaneously, by calculating the L2 loss between the first text encoding, the second text encoding, and the image encoding, the model can further optimize the alignment of text and image features, reducing bias in feature representation. Finally, by combining the total contrastive loss with the L2 loss, the model can comprehensively optimize feature representation at multiple levels, improving the accuracy and generalization ability of the image-text matching task.

[0166] Based on the above embodiments, in a specific scenario embodiment, please refer to... Figure 7 This is a schematic diagram of the architecture of a text-image matching model provided in an embodiment of this application.

[0167] like Figure 7 As shown in the embodiments of this application, the image-text matching model extracts features from images and text through an encoder, and then optimizes the feature representation through contrastive learning to achieve matching between images and text. Wherein:

[0168] The image processing step inputs the original image and the masked image into the image encoder. The image is segmented into multiple regions, and each region is processed by a Transformer block to extract region features. Text descriptions are divided into two groups: a first text description (token count less than or equal to 77) and a second text description (token count greater than 77). The first text description is processed directly by the Transformer block, while the second text description is processed using token embedding and positional encoding extensions for longer text. Then, grouped feature alignment is performed on the image and text features separately, and the total contrastive loss is determined based on the contrastive loss of each group. Finally, the loss of the image-text matching model is determined based on the total contrastive loss, the first L2 loss, and the second L2 loss, and the model is trained based on this loss. This not only improves the model's performance but also enhances its stability and adaptability in complex scenes.

[0169] Please refer to Figure 8 ,for Figure 2 The provided training method for an image-text matching model includes a flowchart illustrating one practical implementation of step 101. Based on the above embodiment, in some possible embodiments, step 101, which involves acquiring multiple sample pairs, may specifically include the following steps:

[0170] In step 701: Obtain the general dataset; the general dataset contains multiple image-text pairs.

[0171] In this embodiment of the application, the general dataset is a collection of data containing multiple image-text pairs, which cover a variety of scenes and topics, providing rich training materials for the model. For example, the general dataset may include datasets such as CC3M, CC12M, and YFCC15M.

[0172] In step 702: The images in the general dataset are re-labeled using a large language model to obtain the corresponding image description text; the number of tokens in the image description text is greater than the second preset value.

[0173] In some possible implementations, a relevant dataset is essential for learning long text information. However, there is currently no suitable general dataset for large-scale training. Therefore, this application uses a large language model (LLAVA-llama2) to relabel the general dataset, extending the text token length to about 120, which is much larger than the original average length of 18 tokens.

[0174] In this embodiment, by inputting images and their original descriptive text from a general dataset into a large language model, the large language model can generate more detailed and richer descriptive text. The number of tokens in these descriptive texts is greater than a preset value, ensuring the completeness and richness of detail in the description. Since the descriptive text in the general dataset is relatively short, it cannot fully express the complex content of the image. The large language model, however, can generate longer and more detailed descriptions, helping the model better understand the image content and thus improving the accuracy of image-text matching.

[0175] In step 703: the relabeled general dataset is subjected to data filtering processing to obtain multiple sample pairs.

[0176] In this embodiment, data filtering is used to remove low-quality or irrelevant image-text pairs, retaining high-quality and relevant sample pairs to improve the quality and applicability of the dataset. Data filtering can improve data quality by removing noisy data and redundant information, thereby increasing the accuracy and reliability of the data. Simultaneously, it reduces the amount of data, lowering the burden of subsequent processing and analysis.

[0177] This application's embodiments utilize a large language model to re-label image description text, generating more detailed and higher-quality descriptions. This overcomes the limitations of short text descriptions and enhances the expressive power of text features. Simultaneously, data filtering further ensures the purity and relevance of the dataset, removing low-quality or irrelevant image-text pairs, thereby improving the quality of the training data.

[0178] In some possible embodiments, after obtaining multiple sample pairs, the following steps may also be performed:

[0179] The sample pairs are divided into multiple datasets of different sizes.

[0180] In this embodiment, by dividing the dataset according to its size, different training strategies can be adopted for datasets of different sizes, thereby improving training efficiency and model performance. Datasets of different sizes can meet different training needs. Small datasets can quickly verify the feasibility of the model, while large datasets can be used for deep training, improving the model's generalization ability. Furthermore, this division method makes the model training process more flexible, allowing for the selection of datasets of appropriate sizes for training and evaluation based on resource and task requirements.

[0181] In some possible implementations, a partitioning strategy can be determined first, such as random partitioning or stratified sampling. Then, the partitioning ratio is set according to task requirements, such as small-scale (e.g., 5M sample pairs), medium-scale (e.g., 15M sample pairs), and large-scale (e.g., 30M sample pairs). Next, sample pairs are drawn from the original dataset to form subsets of different sizes, and the data quality of each subset is verified to ensure data representativeness and diversity. Finally, the partitioned datasets are applied to different training and evaluation phases to ensure consistent model performance across data of different sizes.

[0182] Based on the above technical solution, this application leverages the generalization ability of the CLIP pre-trained model and the rich detail information contained in long texts to improve the alignment ability of the image-text pre-trained model. In long text and short text benchmark tests, this application outperforms state-of-the-art methods by 13% and 5%, respectively. Furthermore, the text encoder in this application can be applied to diffusion models as an effective support. Its retrieval evaluation is shown in Table 1.

[0183]

[0184] Table 1 shows the recall rates for long text image retrieval on the 1k ShareGPT4V and Urban-1k datasets.

[0185] In addition to long text performance, this application can also retain short text performance and the generalization ability of the original CLIP model to the greatest extent. As can be seen from Tables 2 and 3, the model of this application has also improved in short text retrieval and classification:

[0186]

[0187] Table 2 shows the recall rates for short text image retrieval on 5k COCO2017 and 1k Flickr30k.

[0188]

[0189] Table 3. Results of the zero-shot image classification task

[0190] The text encoder in this application can also be directly inserted into Stable-Diffusion-XL for text-to-image tasks. Similarly, by replacing the original CLIP-L text encoder with the method and Long-CLIP model of this application, the model in this application can extract better detail representations and more accurate descriptions. Please refer to... Figure 9 This is a schematic diagram illustrating the effect of a text-based image task provided in an embodiment of this application. In the text-based image task, the text description is: "A shiny gray-blue car is parked on a black cobblestone street, facing left, dappled with sunlight. This is an outdoor, eye-level photograph. The car has large front fenders extending to the doors, as well as large rear fenders edged with light brown wood. The front of the car has a two-seater cockpit, while the rear features a flat roof with light brown wood trim and cargo space for loading items. The tires are black, but the sidewalls are white, while the rims are shiny chrome with a red center." Figure 9 As shown, compared with the existing Long-CLIP model, the image generated by the model in this application has better results.

[0191] Based on the same inventive concept, embodiments of this application also provide a training device for a text-image matching model, the training device comprising:

[0192] The acquisition module is used to acquire multiple sample pairs; each sample pair includes a sample image and corresponding sample input data; the sample input data corresponding to any sample image includes sample description text that describes the image content contained in the sample image;

[0193] The text feature extraction module is used to process the first descriptive text using the first text encoder of the image-text matching model, and to process the second descriptive text using the second text encoder of the image-text matching model, to obtain the corresponding text features. The positional encoding in the first text encoder reuses the original positional encoding of the pre-trained text encoder. The positional encoding of the first n positions in the second text encoder reuses the first n positions in the original positional encoding. The positional encoding of the (n+1)th to mth positions in the second text encoder is obtained by interpolating the encoding of the (n+1)th position and beyond in the original positional encoding. n and m are preset values. The first descriptive text is a sample descriptive text with a token count less than or equal to the first preset value. The second descriptive text is a sample descriptive text with a token count greater than the first threshold.

[0194] The image feature extraction module is used to extract image features from sample images using an image encoder based on an image-text matching model.

[0195] The training module is used to train the image-text matching model based on text features and image features.

[0196] In some possible embodiments, the image feature extraction module is specifically used for:

[0197] For each transformer block in the image encoder, obtain the region cueing parameters corresponding to the transformer block;

[0198] Obtain the initial input sequence for the transformer block;

[0199] The region hint parameters are inserted into the initial input sequence to obtain the final input sequence of the transformer block;

[0200] The final input sequence is fed into the transformer block, which processes the final input sequence to obtain the image features of the sample image.

[0201] In some possible embodiments, the image feature extraction module is specifically used for:

[0202] Obtain the one-way attention mask corresponding to the cue parameters for each region;

[0203] Each unidirectional attention mask is used to mask the image encoding in the final input sequence to obtain the image encoding of each region.

[0204] The image encoding of each region is processed based on a multi-head self-attention mechanism to obtain the image features of each region.

[0205] In some possible embodiments, the training module is specifically used for:

[0206] Calculate the contrast loss between text features and image features, and then calculate the gradient based on the contrast loss;

[0207] The model parameters of the image-text matching model are updated based on the gradient; the model parameters include the region hint parameters corresponding to the transformer block.

[0208] In some possible embodiments, the training module is specifically used for:

[0209] The transformer blocks in the first text encoder, the second text encoder, and the image encoder are grouped for processing; each group contains the same number of transformer blocks.

[0210] Based on the grouping results, the text features and image features are respectively subjected to group feature aggregation, linear transformation and layer normalization to obtain the corresponding text group feature matrix and image group feature matrix.

[0211] Calculate the group contrast loss between the feature matrix of each text group and the feature matrix of each image group;

[0212] The loss of the image-text matching model is determined based on group contrast loss, and the training of the image-text matching model is considered complete when the loss converges.

[0213] In some possible embodiments, the training module is specifically used for:

[0214] Obtain the group loss weight corresponding to each group contrast loss, and determine the total contrast loss by summing the products of each group contrast loss and its corresponding group loss weight;

[0215] Obtain the first text code corresponding to the first descriptive text, the second text code corresponding to the second descriptive text, and the image code corresponding to the sample image;

[0216] Calculate the first L2 loss between the first text encoding and the image encoding, and calculate the second L2 loss between the second text encoding and the image encoding;

[0217] The sum of the first L2 loss, the second L2 loss, and the total contrast loss is determined as the loss of the image-text matching model.

[0218] In some possible embodiments, the acquisition module is specifically used for:

[0219] Obtain a general dataset; the general dataset contains multiple image-text pairs;

[0220] The images in the general dataset are re-labeled using a large language model to obtain corresponding image description text; the number of tokens in the image description text is greater than a second preset value;

[0221] The relabeled general dataset was processed to obtain multiple sample pairs.

[0222] In some possible embodiments, the acquisition module is specifically used for:

[0223] The sample pairs are divided into multiple datasets of different sizes.

[0224] Corresponding to the above embodiments, this application also provides an electronic device. Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 1000 may include a processor 1001, a memory 1002, and a communication unit 1003. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0225] The communication unit 1003 is used to establish a communication channel, enabling the electronic device to communicate with other devices. It receives user data from other devices or sends user data to other devices.

[0226] The processor 1001 serves as the control center of the electronic device, connecting various parts of the device via interfaces and lines. It executes software programs and / or modules stored in the memory 1002 and retrieves data stored in the memory to perform various functions and / or process data. The processor may be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 1001 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0227] The memory 1002 is used to store the execution instructions of the processor 1001. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0228] When the execution instructions in memory 1002 are executed by processor 1001, the electronic device 1000 is able to perform operations. Figure 1 Some or all of the steps in the illustrated embodiments.

[0229] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps of the various embodiments of the image-text matching model training method provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0230] In some possible implementations, various aspects of the terminal device control method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in a training method for a text-image matching model according to various exemplary embodiments of this application as described above.

[0231] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0232] The program product for controlling a terminal device according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0233] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0234] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A training method for an image-text matching model, characterized in that, The method includes: Multiple sample pairs are acquired; each sample pair includes a sample image and corresponding sample input data; the sample input data corresponding to any sample image includes sample description text that describes the image content contained in the sample image; The first descriptive text is processed using a first text encoder of the image-text matching model, and the second descriptive text is processed using a second text encoder of the same model to obtain corresponding text features. The positional encoding in the first text encoder reuses the original positional encoding of the pre-trained text encoder. The positional encoding of the first n positions in the second text encoder reuses the first n positions of the original positional encoding. The positional encoding of the (n+1)th to mth positions in the second text encoder is obtained by interpolating the encoding of the (n+1)th position and beyond in the original positional encoding. n and m are preset values. The first descriptive text is a sample descriptive text with a token count less than or equal to a first preset value. The second descriptive text is a sample descriptive text with a token count greater than a first threshold. The image encoder of the image-text matching model is used to extract image features from the sample image; The image-text matching model is trained based on the text features and the image features.

2. The method according to claim 1, characterized in that, The image encoder of the image-text matching model extracts image features from the sample image, including: For each transformer block in the image encoder, obtain the region cueing parameters corresponding to the transformer block; Obtain the initial input sequence of the transformer block; The region hint parameter is inserted into the initial input sequence to obtain the final input sequence of the transformer block; The final input sequence is input into the transformer block, which processes the final input sequence to obtain the image features of the sample image.

3. The method according to claim 2, characterized in that, The transformer block processes the final input sequence to obtain the image features of the sample image, including: Obtain the one-way attention mask corresponding to each of the region cue parameters; Each of the unidirectional attention masks is used to mask the image codes in the final input sequence to obtain the image codes for each region. The image encoding of each region is processed based on a multi-head self-attention mechanism to obtain the image features of each region.

4. The method according to claim 2, characterized in that, The training of the image-text matching model based on the text features and the image features includes: Calculate the contrast loss between the text features and the image features, and calculate the gradient based on the contrast loss; The model parameters of the image-text matching model are updated according to the gradient; the model parameters include the region hint parameters corresponding to the transformer block.

5. The method according to claim 1, characterized in that, The training of the image-text matching model based on the text features and the image features includes: The transformer blocks in the first text encoder, the second text encoder, and the image encoder are grouped; each group contains the same number of transformer blocks. Based on the grouping results, the text features and the image features are respectively subjected to group feature aggregation, linear transformation and layer normalization to obtain the corresponding text group feature matrix and image group feature matrix. Calculate the group contrast loss between the feature matrix of the text group and the feature matrix of the image group for each group; The loss of the image-text matching model is determined based on the group contrast loss, and the training of the image-text matching model is determined to be complete when the loss converges.

6. The method according to claim 5, characterized in that, The step of determining the loss of the image-text matching model based on the group contrast loss includes: Obtain the group loss weight corresponding to each group contrast loss, and determine the total contrast loss by summing the products of each group contrast loss and its corresponding group loss weight; Obtain the first text code corresponding to the first descriptive text, the second text code corresponding to the second descriptive text, and the image code corresponding to the sample image; Calculate the first L2 loss between the first text encoding and the image encoding, and calculate the second L2 loss between the second text encoding and the image encoding; The sum of the first L2 loss, the second L2 loss, and the total contrast loss is determined as the loss of the image-text matching model.

7. The method according to claim 1, characterized in that, The acquisition of multiple sample pairs includes: Obtain a general dataset; the general dataset contains multiple image-text pairs; The images in the general dataset are re-labeled using a large language model to obtain corresponding image description text; the number of tokens in the image description text is greater than a second preset value; The relabeled general dataset is subjected to data filtering processing to obtain the multiple sample pairs.

8. The method according to claim 7, characterized in that, After obtaining the plurality of sample pairs, the method further includes: The sample pairs are divided into multiple datasets of different sizes.

9. An electronic device, characterized in that, It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method of any one of claims 1-8.

11. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-8.

Images