Training method of CORCLIP model applied to downstream task

Abstract

The invention discloses a CORCLIP model training method applied to downstream tasks. The CORCLIP model training method comprises the following steps: acquiring text data T and corresponding image data I; performing context optimization on the text data T to obtain context vocabularies, and generating language representation ti by utilizing a text encoder in combination with corresponding marks; segmenting a region of interest ROI in the image data I into a plurality of region objects through an attention mechanism, and generating a visual representation vi for the features of the plurality of region objects by using an image encoder; carrying out CORCLIP model learning through cross-modal similarity calculation, and generating a text-to-image corresponding vector pi (T, I) and an image-to-text corresponding vector pi (I, T); and a KL-Divergence overall loss function is constructed, and an optimized CORCLIP model is obtained. The CORCLIP model can be widely applied to downstream tasks such as image subtitle generation, visual question and answer, cross-modal retrieval and the like through context optimization and region-of-interest guidance of an attention mechanism, and particularly improves segmentation precision and semantic understanding accuracy when a key image region is processed.

Description

Technical Field

[0001] The present invention relates to the intersection of computer vision and natural language processing, and more specifically to a training method for a CORCLIP model applied to downstream tasks. Background Art

[0002] With the rapid development of multimedia technology, large-scale visual language pre-training models (such as CLIP, DALL-E, ALIGN, ALBEF, etc.) have demonstrated excellent performance in multiple downstream tasks, including but not limited to image caption generation, visual question answering (VQA), cross-modal retrieval, etc. These pre-training models have acquired rich image-text understanding capabilities through joint training of large-scale image-text pairs, and can achieve efficient migration in a variety of visual language tasks.

[0003] However, despite the significant progress made by existing methods, there are still some limitations. Existing vision-language pre-training models are usually based on a core assumption that the image-text pairs captured on the Internet are in perfect one-to-one correspondence. However, this assumption often does not hold true in practical applications. The correlation between images and texts on the Internet may be weak or even mismatched, which severely limits the generalization ability and accuracy of the model.

[0004] For highly context-sensitive tasks, existing pre-trained models (such as CLIP) require a lot of manual tuning when processing, especially for word selection and tuning. This process is not only time-consuming and labor-intensive, but also a small change in wording may have a huge impact on model performance. Therefore, it is particularly important to develop an automated context optimization mechanism.

[0005] At the same time, image segmentation tasks require the model to accurately identify and segment specific areas in the image. However, when dealing with image segmentation tasks, existing pre-trained models often lack optimization mechanisms for key areas in the image, resulting in low segmentation accuracy. This is mainly because the model does not effectively utilize local information in the image and fails to focus on the area of ​​interest.

[0006] In addition, although models such as CLIP have achieved cross-modal matching of images and texts, the encoding process of image and text data is usually independent of each other, lacking an effective mechanism for information interaction between modalities. This independence makes it difficult to eliminate the heterogeneity between modal data, limiting the model's ability to understand complex multimodal data.

[0007] Moreover, the traditional cross entropy loss function may not fully reflect the difference between the model prediction and the true label when processing multimodal data. Especially in scenarios involving joint training of images and texts, the traditional loss function may not effectively guide the optimization process of the model, resulting in insufficient training or limited model performance.

[0008] Therefore, how to design a training method for the CORCLIP model applied to downstream tasks and improve the performance of the visual language pre-training model in processing complex tasks is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0009] In view of this, the present invention provides a training method for the CORCLIP model applied to downstream tasks, which avoids manual parameter adjustment through an automated mechanism and quickly adapts to the task context; and combines the attention mechanism to accurately extract key image information and improve processing efficiency and accuracy; and adopts KL-Divergence optimization training to enhance the accuracy, generalization ability and robustness of the model's cross-modal similarity calculation, thereby ensuring the stable and efficient performance of the model in multiple tasks.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] A method for training a CORCLIP model for downstream tasks, comprising the following steps:

[0012] S1, obtaining text data T and corresponding image data I;

[0013] S2. Generate text-image embedding pairs (t i ,v i ); specifically including:

[0014] S21, context optimization is performed on the text data T to obtain context vocabulary, and combined with the corresponding tags, a text encoder is used to generate a language representation t i ;

[0015] The region of interest ROI in the image data I is segmented into multiple region objects through the attention mechanism, and the image encoder is used to generate a visual representation v based on the features of multiple region objects. i ;

[0016] S22, expressing the language i and visual representation v i Perform L2 norm normalization to generate text-image embedding pairs (t i ,v i );

[0017] S3, based on the text-image embedding pair (t i ,vi ), CORCLIP model learning is performed through cross-modal similarity calculation to generate the corresponding vector p from text to image i (T,I) and the corresponding vector p from image to text i (I,T);

[0018] S4. Construct the KL-Divergence overall loss function, optimize the learned CORCLIP model, and obtain the optimized CORCLIP model.

[0019] Preferably, in S21, the text encoder comprises a multi-layer Transformer encoder, wherein each text sequence comprises an SOS token and an EOS token and is limited to a fixed length;

[0020] The text encoder is configured to generate a text representation from a natural language, including converting an input text sequence into a lowercase byte pair encoding representation, and generating a text embedding vector after processing through a Transformer.

[0021] Preferably, in said S21, a language representation t is generated i ,include:

[0022] Define a text encoder g, the prompt symbol t of the text encoder g is represented by:

[0023] t=[V1][V2]……[V m ][class]

[0024] Among them, [V1], [V2] and [V m ] represents a vector with the same dimension as the word embedding, m represents a hyperparameter that specifies the number of context tokens, and [class] represents a text category tag;

[0025] Inputting the prompt symbol t into the text encoder g to obtain a classification weight vector representing the visual concept;

[0026] Based on the classification weight vector representing the visual concept, the probability of belonging to each category is calculated:

[0027]

[0028] Among them, p(y=i|x) represents the probability of category y=i for a given input x, sim(·) represents the similarity function, and g(t i ) represents the text encoding result processed by the text encoder, f represents the image feature representation processed by the image encoder, τ is the learnable temperature parameter, N represents the total number of categories, and j represents the category index.

[0029] Preferably, the prompt symbol t of the text encoder g is also expressed as:

[0030] t=[V1][V2]……[class]……[V m-1 ][V m ]

[0031] Among them, the text category tag [class] is placed in the middle of the sequence.

[0032] Preferably, in S22, the image encoder adopts ResNet-50 or visual transformer convolutional neural network architecture;

[0033] The image encoder is configured to receive a high-dimensional image input and map the image into a low-dimensional embedding space; the embedding space dimension output by the image encoder is a fixed value.

[0034] Preferably, in S3, the cross-modal similarity calculation includes:

[0035] S31, calculating image data T i The i-th ROI region and text data I i The similarity probability p between the jth ROI regions in ij (R,R):

[0036]

[0037] Among them, r i Represents the visual representation v i The obtained ROI features, r j Indicates that from the language representation t i The obtained ROI features;

[0038] S32, supervise the correspondence between image and text through ROI self-similarity, and calculate the softened target

[0039]

[0040] Among them, y i is a hard label, indicating the true category; p i (R, R) is a soft label, which indicates the similarity between image and text; β is a mixing coefficient, which is used to balance the influence of hard labels and soft labels.

[0041] Preferably, in S3, a corresponding vector p from text to image is generated i (T,I) and the corresponding vector p from image to text i (I,T), including:

[0042]

[0043] in,

[0044] Preferably, in S4, the KL-Divergence overall loss function is:

[0045]

[0046] Among them, L CLIP =L v2l +L l2v ,

[0047]

[0048] L v2l represents the vision-to-language loss function, L l2v Represents the loss function from language to vision;

[0049] For L' v2l The symmetric function of For L' v ' 2l The symmetric function, L' v2l is the soft loss function from vision to language, L' v ' 2l For L' v2l The relationship enhancement type;

[0050]

[0051] λ and μ are weight parameters.

[0052] Preferably, construct the loss function L' v ' 2l ,include:

[0053] For any distributed vector definition represents the corresponding negative disentanglement distribution, and the element calculation formula of the negative disentanglement distribution is:

[0054]

[0055] Among them, i represents the current image-text pair, j represents other image-text pairs except i, and k represents all image-text pairs;

[0056] Apply the negative unwrapping distribution to the softened target and the corresponding vector p from image to text i (I,T), get L v2l Relationship Enhancement

[0057]

[0058] in, and p i * (I,T) are and p i Negative disentanglement distribution of (I,T).

[0059] Preferably, construct a loss function And the loss function include:

[0060]

[0061]

[0062] The loss function And the loss function Symmetrization is performed by adding a reverse term to the swapped input distributions.

[0063] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0064] Beneficial effects:

[0065] 1. Introducing context optimization, by using continuous vectors learned end-to-end from the data to model context words, while freezing a large number of pre-trained parameters, thus avoiding the problem of manually adjusting prompt words in traditional methods. This not only saves a lot of time and effort, but also improves the flexibility of the model, allowing additional descriptions to be added after the prompt or ending the sentence early.

[0066] 2. Use the attention mechanism to guide ROI for image segmentation tasks, divide the image into different regions or objects, and select the region of interest for processing. This method can focus on the key parts of the image and improve the effect of image segmentation. This segmentation method allows the model to focus on the most relevant parts of the image instead of processing the entire image uniformly.

[0067] 3. Use KL-Divergence to replace the traditional cross entropy as part of the loss function to optimize the overall loss function of the model. Experiments have shown that this approach can stably bring significant improvements, whether in pre-training data of different scales or under different model architectures.

[0068] 4. Model learning is performed through cross-modal similarity calculation to generate corresponding vectors from text to image and image to text, which enhances the correlation between image and text. This is beneficial for eliminating the differences in modal data and enables the model to better understand and process multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0070] Figure 1 A flow chart of a training method for a CORCLIP model applied to downstream tasks provided by an embodiment of the present invention;

[0071] Figure 2 A framework diagram of the CORCLIP model applied to downstream tasks provided by an embodiment of the present invention;

[0072] Figure 3 A flowchart of a process for generating text-image embedding pairs provided by an embodiment of the present invention;

[0073] Figure 4 A schematic diagram of comparing Recall@1 indicators of different models based on the Flickr30 dataset provided by an embodiment of the present invention;

[0074] Figure 5 A schematic diagram of comparing the Recall@5 index of different models based on the Flickr30 dataset provided by an embodiment of the present invention;

[0075] Figure 6 A schematic diagram of comparing the Recall@10 indicators of different models based on the Flickr30 dataset provided by an embodiment of the present invention;

[0076] Figure 7 A schematic diagram of comparing the Recall@1 indicators of different models based on the MSCOCO dataset provided by an embodiment of the present invention;

[0077] Figure 8 A schematic diagram of comparing the Recall@5 indicators of different models based on the MSCOCO dataset provided by an embodiment of the present invention;

[0078] Fig. 9 A schematic diagram of comparing the Recall@10 indicators of different models based on the MSCOCO dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] Embodiment 1;

[0081] like Figure 1 and Figure 2 As shown, this embodiment provides a training method for a CORCLIP model applied to downstream tasks, comprising the following steps:

[0082] S1, obtaining text data T and corresponding image data I;

[0083] S2. Generate text-image embedding pairs (t i ,v i );

[0084] like Figure 3 As shown, specifically including:

[0085] S21, context optimization is performed on the text data T to obtain context vocabulary, and combined with the corresponding tags, a text encoder is used to generate a language representation t i ;

[0086] The region of interest ROI in the image data I is segmented into multiple region objects through the attention mechanism, and the image encoder is used to generate a visual representation v based on the features of multiple region objects. i ;

[0087] S22, expressing the language i and visual representation v i Perform L2 norm normalization to generate text-image embedding pairs (t i ,v i );

[0088] S3, based on the text-image embedding pair (t i ,v i ), CORCLIP model learning is performed through cross-modal similarity calculation to generate the corresponding vector p from text to image i (T,I) and the corresponding vector p from image to text i (I,T);

[0089] S4. Construct the KL-Divergence overall loss function, optimize the learned CORCLIP model, and obtain the optimized CORCLIP model.

[0090] The CORCLIP model is guided by the region of interest of context optimization and attention mechanism, and can be widely used in downstream tasks such as image caption generation, visual question answering, cross-modal retrieval, etc., especially when processing key image areas, it improves the segmentation accuracy and semantic understanding accuracy.

[0091] The above steps and related features are further described in detail below:

[0092] In this embodiment S1, text data T and corresponding image data I are obtained;

[0093] Specifically, text data usually contains text information describing the content of an image, such as the title, description, or tags of an image. Image data is the image corresponding to these text descriptions. In order to ensure that the CORCLIP model can effectively learn the correspondence between images and text, this data must have a certain degree of relevance and representativeness.

[0094] In this embodiment S2, the CORCLIP model is used to generate a text-image embedding pair (t i ,v i ); specifically including:

[0095] S21, context optimization is performed on the text data T to obtain context vocabulary, and combined with the corresponding tags, a text encoder is used to generate a language representation t i ;

[0096] The text encoder consists of a multi-layer Transformer encoder, where each text sequence contains SOS tokens and EOS tokens and is limited to a fixed length;

[0097] The text encoder is configured to generate a text representation from a natural language, including converting an input text sequence into a lowercase byte pair encoding representation, and generating a text embedding vector after processing through a Transformer.

[0098] Generate language representation i ,include:

[0099] Define a text encoder g, the prompt symbol t of the text encoder g is represented by:

[0100] t=[V1][V2]……[V m ][class]

[0101] Among them, [V1], [V2] and [V m ] represents a vector with the same dimension as the word embedding, m represents a hyperparameter that specifies the number of context tokens, and [class] represents a text category tag;

[0102] Inputting the prompt symbol t into the text encoder g to obtain a classification weight vector representing the visual concept;

[0103] Based on the classification weight vector representing the visual concept, the probability of belonging to each category is calculated:

[0104]

[0105] Among them, p(y=i|x) represents the probability of category y=i for a given input x, sim(·) represents the similarity function, and g(t i ) represents the text encoding result processed by the text encoder, f represents the image feature representation processed by the image encoder, τ is the learnable temperature parameter, N represents the total number of categories, and j represents the category index.

[0106] In addition, the prompt symbol t of the text encoder g is also expressed as:

[0107] t=[V1][V2]……[class]……[V m-1 ][V m ]

[0108] Among them, the text category tag [class] is placed in the middle of the sequence, which increases the flexibility of learning.

[0109] Specifically, the prompt can fill the following cell with supplementary description, or cut off the sentence early by using a termination signal (such as a period).

[0110] Furthermore, the region of interest ROI in the image data I is segmented into multiple region objects through the attention mechanism, and the image encoder is used to generate a visual representation v for the features of the multiple region objects. i ;

[0111] The image encoder adopts a ResNet-50 or visual transformer convolutional neural network architecture; the image encoder is configured to receive a high-dimensional image input and map the image into a low-dimensional embedding space; the embedding space dimension output by the image encoder is a fixed value.

[0112] S22, expressing the language i and visual representation v i Perform L2 norm normalization to generate text-image embedding pairs (t i ,v i );

[0113] In step S2, the CORCLIP model first processes the text data through the context optimization technique and generates a language representation using a text encoder containing a multi-layer Transformer encoder. Then, the attention mechanism is used to segment the region of interest (ROI) in the image data and a visual representation is generated through the image encoder. Finally, the generated language representation and visual representation are normalized by L2 norm to generate a text-image embedding pair.

[0114] It improves the model's ability to capture the correlation between text and images and enhances the model's expressiveness in multimodal tasks. In particular, through context optimization and ROI processing, the model can more accurately understand and match text and images in complex scenarios, thereby improving the accuracy and robustness of cross-modal tasks.

[0115] In this embodiment S3, based on the text-image embedding pair (t i ,v i ), CORCLIP model learning is performed through cross-modal similarity calculation to generate the corresponding vector p from text to image i (T,I) and the corresponding vector p from image to text i (I,T);

[0116] Specifically, the cross-modal similarity calculation includes:

[0117] S31, calculating image data T i The i-th ROI region and text data I i The similarity probability p between the jth ROI regions in ij (R,R):

[0118]

[0119] Among them, r i Represents the visual representation v i The obtained ROI features, r j Indicates that from the language representation t i The obtained ROI features;

[0120] S32, supervise the correspondence between image and text through ROI self-similarity, and calculate the softened target

[0121]

[0122] Among them, y i is a hard label, indicating the true category; p i (R, R) is a soft label, which indicates the similarity between image and text; β is a mixing coefficient, which is used to balance the influence of hard labels and soft labels.

[0123] Further, generate the corresponding vector p from text to image i (T,I) and the corresponding vector p from image to text i (I,T), including:

[0124]

[0125] in,

[0126] In this step, the correspondence between them is measured by calculating the similarity probability between the corresponding regions of interest (ROI) in the image and text. It not only considers the matching degree between different parts of the image and the text description, but also constructs a more stable training target through the weighted average of hard labels and soft labels, thereby improving the performance of the model in cross-modal retrieval tasks.

[0127] In this embodiment S4, a KL-Divergence overall loss function is constructed to optimize the learned CORCLIP model to obtain an optimized CORCLIP model;

[0128] The overall loss function of KL-Divergence is:

[0129]

[0130] Among them, L CLIP =L v2l +L l2v ,

[0131]

[0132] L v2l represents the vision-to-language loss function, L l2v Represents the loss function from language to vision;

[0133] For L' v2l The symmetric function of For L' v ' 2l The symmetric function, L' v2l is the soft loss function from vision to language, L' v ' 2l For L' v2l The relationship enhancement type;

[0134]

[0135] λ and μ are weight parameters.

[0136] Specifically, construct the loss function L' v '2l ,include:

[0137] For any distributed vector definition represents the corresponding negative disentanglement distribution, and the element calculation formula of the negative disentanglement distribution is:

[0138]

[0139] Among them, i represents the current image-text pair, j represents other image-text pairs except i, and k represents all image-text pairs;

[0140] Apply the negative unwrapping distribution to the softened target and the corresponding vector p from image to text i (I,T), get L v2l Relationship Enhancement

[0141]

[0142] in, and p i * (I,T) are and p i Negative disentanglement distribution of (I,T).

[0143] Specifically, construct the loss function And the loss function include:

[0144]

[0145] The loss function And the loss function Symmetrization is performed by adding a reverse term to the swapped input distributions.

[0146] In this step, the loss function is constructed using the symmetric KL-Divergence. By symmetrizing the loss functions in the vision-to-language and language-to-vision directions, the stability of model training is enhanced. By introducing negative disentanglement distribution to optimize the softened target and the corresponding vector, the relationship between the modalities is further strengthened. This effectively reduces the unstable factors in the training process, making the CORCLIP model more robust and more generalizable when processing multimodal data.

[0147] In this embodiment, the CORCLIP model is optimized through multi-step training so that it can efficiently handle downstream tasks. First, collect text and corresponding image data. Then, use a text encoder (containing multiple layers of Transformer to process text sequences to generate text embeddings) and an image encoder (based on ResNet-50 or ViT to map images to low-dimensional embedding space) to extract feature representations of text and images respectively.

[0148] In the specific training process, the attention mechanism is combined to calculate the cross-modal similarity between text and image, and the generated text embedding and image embedding are semantically aligned through joint learning. In particular, the method introduces context optimization and ROI (region of interest) self-similarity supervision to enhance the model's understanding of context and recognition of key areas of the image. Finally, the KL-Divergence loss function is used to optimize the model, and the difference between the model prediction and the true label is more comprehensively evaluated by considering the symmetry of the distribution, thereby obtaining the optimized CORCLIP model.

[0149] Embodiment 2;

[0150] In this embodiment, the optimized CORCLIP model is applied to specific downstream tasks, and its application process is further described in detail through image caption generation, including:

[0151] 1) Preparation stage;

[0152] Collect a dataset containing a large number of images and their corresponding text descriptions. These datasets usually include the title, description or tags of the image, which are used to train the CORCLIP model. Load the CORCLIP model that has been optimized by the above training method, including the text encoder and image encoder, as well as the model parameters.

[0153] 2) Input processing;

[0154] The image to be captioned is fed into the image encoder. The image encoder extracts features from the image and maps them to a low-dimensional embedding space. At the same time, in order to generate captions for the image, an initial text prompt or an empty text sequence is defined as input to the text encoder. This helps the model have a starting point when generating captions.

[0155] 3) Generate text-image embedding pairs;

[0156] The text encoder receives a text prompt (or an empty sequence) and generates a language representation of the text through a multi-layer Transformer encoder. In this process, context optimization is used to generate a more descriptive language representation. The image encoder processes the input image, uses the attention mechanism to focus on the region of interest (ROI) in the image, and generates a visual representation of these regions.

[0157] The generated language representation and visual representation are L2-norm normalized to form a text-image embedding pair.

[0158] 4) Subtitle generation

[0159] Based on the text-image embedding pairs, the cross-modal similarity between the text representation and the image representation is calculated. In this way, the model can identify the text description that is most relevant to the image content.

[0160] The Transformer-based decoder decodes the text sequence from the text representation as the caption of the image. During the decoding process, the model predicts the next word based on the previously generated words and the current text-image embedding pair until it reaches a predetermined length or encounters a specific end symbol (such as EOS).

[0161] The generated captions are compared with the real text description to evaluate their accuracy and relevance. This is done by calculating the similarity between the generated captions and the real description (such as BLEU score or ROUGE score).

[0162] The evaluation results are fed back to the CORCLIP model for further training and optimization, including fine-tuning model parameters or adjusting strategies during the generation process to improve the accuracy and naturalness of caption generation.

[0163] Through the above steps, the optimized CORCLIP model can be effectively applied to the image caption generation task, and can generate accurate and descriptive captions by combining the feature representations of text and images. At the same time, the model has high segmentation accuracy and semantic understanding accuracy when processing key image areas, further improving the quality of generated captions.

[0164] In addition, this embodiment also uses the data set Flickr30 and the data set MSCOCO, and combines the evaluation indicator Recall@k to compare the specific performance of the optimized CORCLIP model in this technical solution with other models on the Image2Text and Text2Image tasks.

[0165] The Flickr30K dataset collects a total of 31,000 images and 155,000 sentences, and is divided into a training set (29,000 images / 145,000 sentences), a validation set (1,000 images / 5,000 sentences), and a test set (1,000 images / 5,000 sentences). The MSCOCO dataset consists of pictures of everyday scenes in everyday environments. It contains a total of 113,287 images and 616,425 sentences, divided into a training set (29,000 images / 145,000 sentences), a validation set (5,000 images / 25,000 sentences), and a test set (5,000 images / 250,000 sentences).

[0166] like Figure 4 , Figure 5 and Figure 6 The comparison results of Recall@k index based on the Flickr30 dataset, as well as Figure 7 , Figure 8 and Fig. 9 The comparison results of the Recall@k index based on the MSCOCO dataset include five models: ImageBERT, CLIP, CoOpCLIP, ALIGN, and the optimized CORCLIP. Specifically, Recall@1 indicates the correct proportion of the first result returned by the system. Recall@5 indicates the correct proportion of the first five results returned by the system. Recall@10 indicates the correct proportion of the first ten results returned by the system.

[0167] Through the above comparison results, the optimized CORCLIP model in this technical solution is overall better than other models in Image2Text and Text2Image tasks, especially in Text2Image tasks, with obvious advantages in Recall@1, Recall@5, and Recall@10.

[0168] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0169] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a CORCLIP model for downstream tasks, characterized in that: The following steps are involved: S1, obtaining text data T and corresponding image data I; S2. Generate text-image embedding pairs (t i ,v i ); specifically including: S21, context optimization is performed on the text data T to obtain context vocabulary, and combined with the corresponding tags, a text encoder is used to generate a language representation t i ; The region of interest ROI in the image data I is segmented into multiple region objects through the attention mechanism, and the image encoder is used to generate a visual representation v based on the features of multiple region objects. i ; S22, expressing the language i and visual representation v i Perform L2 norm normalization to generate text-image embedding pairs (t i ,v i ); S3, based on the text-image embedding pair (t i ,v i ), CORCLIP model learning is performed through cross-modal similarity calculation to generate the corresponding vector p from text to image i (T,I) and the corresponding vector p from image to text i (I,T); S4. Construct the KL-Divergence overall loss function, optimize the learned CORCLIP model, and obtain the optimized CORCLIP model.

2. A method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S21, the text encoder includes a multi-layer Transformer encoder, wherein each text sequence contains SOS tokens and EOS tokens and is limited to a fixed length; The text encoder is configured to generate a text representation from a natural language, including converting an input text sequence into a lowercase byte pair encoding representation, and generating a text embedding vector after processing through a Transformer.

3. The method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S21, a language representation t is generated. i ,include: Define a text encoder g, the prompt symbol t of the text encoder g is represented by: t=[V1][V2]……[V m ][class] Among them, [V1], [V2] and [V m ] represents a vector with the same dimension as the word embedding, m represents a hyperparameter that specifies the number of context tokens, and [class] represents a text category tag; Inputting the prompt symbol t into the text encoder g to obtain a classification weight vector representing the visual concept; Based on the classification weight vector representing the visual concept, the probability of belonging to each category is calculated: Among them, p(y=i|x) represents the probability of category y=i for a given input x, sim(·) represents the similarity function, and g(t i ) represents the text encoding result processed by the text encoder, f represents the image feature representation processed by the image encoder, τ is the learnable temperature parameter, N represents the total number of categories, and j represents the category index.

4. A method for training a CORCLIP model for downstream tasks according to claim 3, characterized in that: The prompt symbol t of the text encoder g is also represented as: t=[V1][V2]......[class]......[V m-1 ][V m ] Among them, the text category tag [class] is placed in the middle of the sequence.

5. The method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S22, the image encoder adopts ResNet-50 or visual transformer convolutional neural network architecture; The image encoder is configured to receive a high-dimensional image input and map the image into a low-dimensional embedding space; the embedding space dimension output by the image encoder is a fixed value.

6. The method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S3, the cross-modal similarity calculation includes: S31, calculating image data T i The i-th ROI region and text data I i The similarity probability pi between the jth ROI regions in j (R, R): Among them, r i Represents the visual representation v i The obtained ROI features, r j Indicates that from the language representation t i The obtained ROI features; S32, supervise the correspondence between image and text through ROI self-similarity, and calculate the softened target Among them, y i is a hard label, indicating the true category; p i (R, R) is a soft label, which indicates the similarity between image and text; β is a mixing coefficient, which is used to balance the influence of hard label and soft label.

7. The method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S3, the corresponding vector p from text to image is generated i (T, I) and the corresponding vector p from image to text i (I, T), including: in, 8. The method for training a CORCLIP model for downstream tasks according to claim 1, characterized in that: In S4, the KL-Divergence overall loss function is: Among them, L CLIP =L v2l +L l2v , L v2l represents the vision-to-language loss function, L l2v Represents the loss function from language to vision; L′ v2l The symmetric function of L″ v2l The symmetric function, L′ v2l is the soft loss function from vision to language, L″ v2l L′ v2l The relationship enhancement type; λ and μ are weight parameters.

9. A method for training a CORCLIP model for downstream tasks according to claim 8, characterized in that: Construct the loss function L' v ' 2l ,include: For any distributed vector definition represents the corresponding negative disentanglement distribution, and the element calculation formula of the negative disentanglement distribution is: Among them, i represents the current image-text pair, j represents other image-text pairs except i, and k represents all image-text pairs; Apply the negative unwrapping distribution to the softened target and the corresponding vector p from image to text i (I,T), get L v2l Relationship Enhancement in, and p i * (I,T) are and p i Negative disentanglement distribution of (I,T).

10. A method for training a CORCLIP model for downstream tasks according to claim 8 or 9, characterized in that: Constructing the loss function And the loss function include: The loss function And the loss function Symmetrization is performed by adding a reverse term to the swapped input distributions.

Images