Vision-language pre-training general framework for realizing multi-granularity cross-modal alignment

Abstract

The invention discloses a vision-language pre-training general framework for realizing multi-granularity cross-modal alignment. Image-text pairs are used as training data to be respectively input into an image encoder and a text encoder to obtain coarse-granularity global features and fine-granularity local features; multi-granularity cross-modal alignment is carried out on the obtained global features and local features, and the aligned global features of the image and the text are used for cross-modal pre-training; the multi-granularity cross-modal alignment is composed of a text perception image block selection module and a cross-granularity alignment module; a text perception image block selection module in the step S3 selects the image block with the highest correlation through the correlation between text words or phrases and the image blocks to form a corresponding region or target object, so that more sufficient fine-grained alignment is realized; the cross-modal pre-training target comprises the multi-granularity cross-modal pre-training target constructed in the step S6 and universal image-text comparison loss, image-text matching loss and mask text modeling loss.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a general vision-language pre-training framework (method) for realizing multi-granularity cross-modal alignment. Background Art

[0002] In recent years, with the rapid development of technology, the way of perceiving the world by a single mode has become outdated. Visual language tasks, as a bridge between computer vision and natural language processing (e.g., visual question answering, image captioning, image text retrieval), have become a hot research direction. The core idea of ​​visual language tasks is to transform visual language tasks into image-text matching tasks, which requires alignment of two different modalities and cross-modal representation learning to improve the reasoning ability of downstream tasks. Although existing models have established interactions between the two modalities, most of these methods focus on coarse-grained alignment and target detection tasks, and there are relatively few research works exploring multi-granular cross-modal alignment methods, which leads to insufficient reasoning capabilities for downstream tasks that have a large demand for fine-grained information mining and multi-granular information fusion, which needs further exploration and research.

[0003] CN202211451088.X discloses a method for grasping target objects in cluttered scenes based on joint vision-language-action modeling. The method processes object-centered representations through pre-trained vision-language models and grasping models to obtain visual-language features and grasping features of each bounding box, and uses transformers to implement a cross-attention mechanism between vision-language-action multimodalities to generate vision-language-action cross-attention features, and then generates and executes decisions, thereby achieving higher sample utilization and avoiding additional data collection and training in the simulation-to-physical migration process. Compared with the two-stage strategy, there is no need to manually design visual attributes for language-vision matching and screening rules for the planner, so that it can adapt to more flexible language instructions and achieve better task generalization.

[0004] CN202211238000.6 Visual language navigation method and application based on cross-modal semantic alignment pre-training, 1. Pre-train the agent by masking and reconstructing a random fragment in the instruction information; 2. Pre-train the agent by extracting discriminative phrases from the instruction information and aligning them with the trajectory fragments; 3. Pre-train the agent using the two proposed pre-training tasks and use imitation learning and reinforcement learning to fine-tune the overall network structure to obtain the optimal model. Mining the potential fine-grained semantic alignment relationship in the trajectory-instruction pair enables the agent to better complete the navigation task. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a general framework for vision-language pre-training to achieve multi-granularity cross-modal alignment.

[0006] Technical solution of the present invention: To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a general framework (method) for visual-language pre-training to achieve multi-granularity cross-modal alignment, comprising the following steps:

[0007] Step S1, using image-text data pairs as training data to input into image encoder and text encoder respectively to obtain global features (coarse-grained) and local features (fine-grained);

[0008] Step S2, performing multi-granularity cross-modal alignment on the global features and local features obtained in step S1, and using the aligned global features and local features of the image and text for cross-modal pre-training;

[0009] Step S3: The multi-granularity cross-modal alignment in step S2 is completed by the text-aware image block selection module and the cross-granularity alignment module;

[0010] Step S4: The text-aware image block selection module in step S3 selects the image block with the highest correlation according to the correlation between the text words or phrases and the image blocks to form the corresponding area or target object, thereby achieving more adequate fine-grained alignment;

[0011] The cross-granularity alignment module in step S5 and step S3 respectively "cross-aligns" the local features of the text with the global features of the image and the local features of the image with the global features of the text, thereby further fusing multi-granularity information while realizing fine-grained information mining;

[0012] Step S6, the text-aware image block selection module in step S4 and the cross-granularity alignment module in step S5 jointly construct a multi-granularity cross-modal pre-training target;

[0013] The cross-modal pre-training objectives in step S7 and step S2 include the multi-granularity cross-modal pre-training objectives constructed in step S6 as well as the general image-text contrast loss, image-text matching loss and masked text modeling loss.

[0014] Furthermore, in step S1, image-text pairs are used as training data D=(V i ,T i ), where each image sample V i is divided into image blocks {p1,p2,p3,...,p N} and image block with p cls Input the image encoder to get the image global feature V g and local image features V l={v1,v2,v3,...,v N}, and each text sample is segmented into {w1,w2,w3,...,w N} and with w cls Input text encoder to get text global feature T g and local image features T l ={t1,t2,t3,...,t N}.

[0015] Furthermore, the global features and local features obtained in step S1 are used to implement multi-granularity cross-modal alignment in step S2.

[0016] Furthermore, the multi-granularity cross-modal alignment in step S2 is composed of a text-aware image block selection module and a cross-granularity alignment module.

[0017] Furthermore, the text-aware image block selection module in step S4 selects the image block with the highest correlation according to the correlation between the text words or phrases and the image blocks to form the corresponding area or target object, and screens out the image local features and text local features corresponding to the image blocks and words rich in important information in the image and text respectively based on the self-attention mechanism, and Represented by V, and together with the image and text global features, constitute the output of the text-aware image patch selection module. s and T s On this basis, the cross-attention mechanism is further used to select the local image features that are most relevant to the local text features.

[0018]

[0019] Its essence is to select the local features of the image with the highest correlation with the local features of the text through the attention feature map, so as to construct the area or target object corresponding to the local features of the text to achieve more sufficient fine-grained alignment:

[0020]

[0021] Furthermore, the cross-granularity alignment module in step S5 cross-aligns the text / image local features with the global features of another modality, and according to the image and text global features V obtained in step S1, g and T g And the local features selected in step S4 and Compute the similarity of cross-granular alignment:

[0022]

[0023] Where S I-W represents the similarity between each image sample and the words in each group of text samples, S T-P Represents the similarity between each text sample and the image block in each group of image samples, so as to calculate the cross-granularity similarity through the inner product:

[0024]

[0025] s global (I,T)=s I-W +s T-P

[0026] Furthermore, the multi-granularity cross-modal pre-training objective constructed in step S6 further mines fine-grained information while fusing multi-granularity information. The multi-granularity similarity between image and text is expressed as:

[0027] s mlga (I,T)=αs global (I,T)+(1-α)s local (I,T)

[0028] The multi-granularity cross-modal pre-training objective is expressed as:

[0029]

[0030] Where (I,T) represents the matched image-text pair, (I,T k ) and (I k ,T) represent the unmatched image-text pairs, η represents the threshold, and the default value is set to 0.2.

[0031] Furthermore, the cross-modal pre-training target constructed in step S7 includes, in addition to the multi-granular cross-modal pre-training target, a general image-text contrast loss, an image-text matching loss, and a masked text modeling loss. g and T g , the image-text similarity s(I,T) can be expressed as V g (T g ) T , the text-image similarity and image-text similarity are expressed as:

[0032]

[0033] Where τ represents the temperature parameter, and the image-text contrast loss is expressed by the cross entropy function:

[0034]

[0035] The global features of the image and the global features of the text are used as input to obtain cross-modal representation through the cross-modal fusion encoder to determine whether the image and the text match. The image-text matching loss is expressed as:

[0036]

[0037] The masked text modeling loss is expressed as:

[0038]

[0039] in Represents the probability of the model's prediction for each mask tag, and the final complete loss function is expressed as:

[0040]

[0041] Beneficial effects: The general vision-language pre-training framework for realizing multi-granular cross-modal alignment provided by the present invention is applied to vision-language tasks, and enhances the cross-modal representation learning ability through fine-grained information mining and multi-granular information fusion, thereby improving the downstream task model reasoning ability. The results of four vision-language downstream tasks show that the algorithm proposed by the present invention is efficient and superior to similar algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This paper compares the differences between the general framework of vision-language pre-training and existing methods for achieving multi-granularity cross-modal alignment, including (a) existing global-lever alignment, (b) existing object alignment, and (c) the multi-granularity alignment proposed by the present invention.

[0043] Figure 2 It is a general framework diagram for visual-language pre-training to achieve multi-granularity cross-modal alignment;

[0044] Figure 3 It is a schematic diagram of the information-based image block word filtering module and the text-aware image block selection module.

[0045] Figure 4 , Figure 5 Specific results of effectiveness and robustness in accurately classifying the relationship between images and texts are shown respectively. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention provides a general framework for visual-language pre-training to achieve multi-granularity cross-modal alignment, comprising the following steps:

[0048] Step S1: Input the image encoder and text encoder to obtain global features (coarse-grained) and local features (fine-grained):

[0049] Specifically, image-text related data pairs are used as training data D = (V i ,T i ), where each image sample V i is divided into image blocks {p1,p2,p3,...,p N}, each image block is converted into a one-dimensional vector and linearly transformed to generate the corresponding image block embedding and combined with the category embedding p cls The 12-layer Vision Transformer (ViT) model is input together, the image classification mark Pcls, and the following Wcls is the text classification mark to generate the image global feature V g and local image features V l ={v1,v2,v3,...,v N}, the specific implementation of the image encoder is as follows:

[0050] 1) The input image is segmented into 16×16 image blocks and mapped to a 768-dimensional feature space through linear transformation;

[0051] 2) Add position encoding to preserve spatial information;

[0052] 3) Use the ViT model (ViT stands for Vision Transformer) to process the image block sequence and extract the global and local features of the image;

[0053] At the same time, each text sample is segmented into a subword sequence {w1,w2,w3,...,w N}, each subword is given three types of embeddings: word embedding, paragraph embedding, and position embedding. The generated embedding vector sequence represents the input text and is compared with w cls The 6-layer Transformer model is used as input to obtain the global feature T of the text. g and text local features T l ={t1,t2,t3,...,t N}, the specific implementation of the text encoder is as follows:

[0054] 1) Use the first 6 layers of the BERT model to process text sequences; the BERT model is a deep bidirectional language representation model based on the Transformer architecture. Through pre-training on massive text data, it learns rich contextual representations and can simultaneously learn contextual information from both the left and right directions of the text, thereby more accurately representing the semantic information in the text.

[0055] 2) Generate global and local features of text.

[0056] Step S2: Implement the multi-granularity cross-modal alignment in step S2 on the global features and local features obtained in step S1, and use the aligned global features and local features of the image and text for cross-modal pre-training; the specific process is as follows:

[0057] 1) Coarse-grained alignment: align the global features of the image and text; calculate the image global feature V g With the global feature T g similarity; that is, S(I,T)=V g (T g ) T

[0058] 2) Fine-grained alignment: The most informative words are selected from the text through the text-aware image block selection module, and the correlation between them and the image blocks is calculated to select the most relevant image blocks; these image blocks and text words together constitute local feature pairs; that is, The similarity between each text word and the image block with the highest correlation selected by the image block selection module is calculated, and the model is trained by maximizing the similarity between them, which is the fine-grained alignment in this paper;

[0059] Step S3, the text-aware image block selection module in step S2 selects the image block with the highest correlation through the correlation between the text words or phrases and the image blocks to form the corresponding area or target object, and screens out the image local features and text local features corresponding to the image blocks and words rich in important information in the image and text based on the self-attention mechanism, and Represented by V, and together with the image and text global features, constitute the output of the text-aware image patch selection module. s and T s Representation; on this basis, the cross attention mechanism is further used to select the local image features that are most relevant to the local text features

[0060]

[0061] Its essence is to select the local features of the image with the highest correlation with the local features of the text through the attention feature map, so as to construct the area or target object corresponding to the local features of the text to achieve more sufficient fine-grained alignment:

[0062]

[0063] A cross represents the cross-modal attention feature map used to select the most relevant image patches in the visual modality based on the textual tags, S represents the query vector generated by n text tokens after passing through the shared query linear layer in the cross-modal self-attention mechanism, and d represents the embedding vector dimension. local (I, T) is the fine-grained similarity calculation in the above fine-grained alignment;

[0064] Step S4: The cross-granularity alignment module in step S3 respectively cross-aligns the local features of the text with the global features of the image and the local features of the image with the global features of the text, and the global features of the text according to the global features of the image and text obtained in step S1. g and T g And the local features selected in step S4 and Compute the similarity of cross-granular alignment:

[0065]

[0066] Where S I-W represents the similarity between each image sample and the words in each group of text samples, S T-P Represents the similarity between each text sample and the image block in each group of image samples, so as to calculate the cross-granularity similarity through the inner product:

[0067]

[0068] sglobal(I,T)=s I-W +s T-P

[0069] Step S5: construct a multi-granularity cross-modal pre-training target to further mine fine-grained information and fuse multi-granularity information. The multi-granularity similarity between image and text is expressed as:

[0070] s mlga (I,T)=αs global (I,T)+(1-α)s local (I,T)

[0071] The multi-granularity cross-modal pre-training objective is expressed as:

[0072]

[0073] Where (I,T) represents the matched image-text pair, (I,T k ) and (I k ,T) represent the unmatched image-text pairs, η represents the threshold, and the default value is set to 0.2. T-P ,s I-W is the weighted sum of the similarities between the image and the words in the selected sentence, S I-W (1,i) is the similarity between the image and the i-th word, S T-P is the weighted sum of the similarities between the sentence and the selected image patches in the image, S T-P (i,1) is the similarity between the sentence and the i-th image patch; s mlga (I,T)

[0074] Indicates the similarity of positive sample pairs, the larger the better, s mlga (I,T k ) and s mlga (I k ,T) represents the similarity of negative sample pairs, which should be as small as possible;

[0075] Step S6, the text-aware image block selection module in step S4 and the cross-granularity alignment module in step S5 jointly construct a multi-granularity cross-modal pre-training target;

[0076] Step S7, the cross-modal pre-training target in step S2 includes the multi-granularity cross-modal pre-training target constructed in step S6 as well as the general image-text contrast loss, image-text matching loss and masked text modeling loss.

[0077] In addition to the multi-granular cross-modal pre-training objectives, the cross-modal pre-training objectives also include the general image-text contrast loss, image-text matching loss, and masked text modeling loss. According to the image and text global features V obtained in step S1, g and T g , the image-text similarity s(I,T) can be expressed as V g (T g ) T , the text-image similarity and image-text similarity are expressed as:

[0078]

[0079] Where τ represents the temperature parameter, and the image-text contrast loss is expressed by the cross entropy function:

[0080] represents the global features of the image, T g Represents the global features of the text

[0081] p i2t(I) represents the similarity distribution between an image and a batch of texts, which is a probability distribution of image-to-text similarity, p t2i (T) represents the similarity distribution between text and images in a batch, which is the probability distribution of similarity from text to image. k and I k Represent negative samples of text and image respectively.

[0082] y i2t (I),p i2t (I) represents the probability distribution of the true label and predicted label of image I in the image-to-text direction, yt2i(T), pt2i(T) represent the probability distribution of the true label and predicted label of text T in the text-to-image direction. H represents the cross entropy loss function

[0083] The global features of the image and the global features of the text are used as input to obtain cross-modal representation through the cross-modal fusion encoder to determine whether the image and the text match. The image-text matching loss is expressed as:

[0084]

[0085] The masked text modeling loss is expressed as:

[0086]

[0087] in Represents the probability of the model's prediction for each mask tag, and the final complete loss function is expressed as:

[0088]

[0089] This is the pre-training loss function, which is used to train the sample pairs in the pre-training data set. The first three Ls are used to prove whether the sample pairs match. Different measurement criteria are designed according to different task types in the prediction stage. In the above steps and during the pre-training process, the model has mastered the rich semantic relationships and multi-granular alignment information between visual and language modalities by learning large-scale image-text pairing data. This pre-trained knowledge can be effectively transferred to various downstream tasks, including image annotation, image-text retrieval, and visual question answering. The following are specific application examples of pre-trained models in downstream tasks:

[0090] Image Annotation:

[0091] 1) Model input: Input an unlabeled image.

[0092] 2) Feature extraction: Use the pre-trained Vision Transformer (ViT) model to extract global and local features of the image.

[0093] 3) Text generation: The extracted image features are input into a pre-trained text generation module to generate natural language text describing the image content.

[0094] 4) Result output: output the annotated text of the image.

[0095] By pre-training on large-scale image-text data, the model has learned how to extract meaningful information from image features and generate corresponding descriptions. This enables the model to perform well in image annotation tasks and generate accurate and detailed image descriptions.

[0096] Image and text search:

[0097] 1) Model input: Input a query text or image.

[0098] 2) Feature extraction: Use a pre-trained model to extract features of the query text or image.

[0099] 3) Similarity calculation: Perform multi-granular similarity calculation on the query features and the features of the candidate images or texts.

[0100] 4) Result sorting: Sort the candidate results according to the similarity score and select the most relevant images or texts as the retrieval results.

[0101] 5) Result output: Output the image or text most relevant to the query.

[0102] The pre-trained model can efficiently match cross-modal data in image-text retrieval tasks and accurately find the image or text that is most relevant to the query. This is because the model has learned a large number of correspondences between images and texts during the pre-training process.

[0103] Visual Question Answering:

[0104] 1) Model input: Input a dialogue containing an image and a question.

[0105] 2) Feature extraction: Use the pre-trained model to extract the visual features of the image and the text features of the question.

[0106] 3) Cross-modal reasoning: The model’s cross-modal reasoning capability is used to combine the features of the image and the question to generate answers.

[0107] 4) Result output: Output the natural language answer to the question.

[0108] In visual question answering tasks, the pre-trained model can understand the relationship between image content and question semantics through cross-modal reasoning capabilities, thereby generating accurate answers. This is due to the model's learning of the complex semantic relationship between images and texts during the pre-training stage.

[0109] Implementation steps:

[0110] 1) Loading pre-trained models: In downstream tasks, first load the model parameters pre-trained on large-scale data.

[0111] 2) Feature extraction: For given input data (image or text), use the pre-trained model to extract the corresponding feature representation.

[0112] 3) Task-specific layer: Add task-specific output layer based on the pre-trained model. For example, for image annotation tasks, you can add a text generation decoder; for image-text retrieval tasks, you can add a similarity calculation module. The global feature V in step S2 g With the global feature T g similarity;

[0113] 4) Fine-tuning: Use data from downstream tasks to fine-tune the model so that it better adapts to the needs of specific tasks.

[0114] 5) Prediction and Evaluation: Use the fine-tuned model to make predictions in downstream tasks and evaluate the performance of the model.

[0115] Through the above steps, the pre-trained model can show excellent performance in a variety of downstream tasks, significantly improving the processing effect and accuracy of the task. The combination of pre-training and fine-tuning not only fully utilizes the advantages of large-scale pre-training data, but also enables the model to quickly adapt to different specific task requirements.

[0116] In order to more comprehensively demonstrate the effect of the improved invention of the visual-language pre-training model in the visual-language downstream tasks, the present invention conducted experiments on four visual-language downstream tasks and compared them with the current optimal algorithms. In the cross-modal retrieval task, our OMEGA model was evaluated on the MSCOCO and Flickr30K datasets. On the Flickr30K dataset, the model achieved R@1, R@5 and R@10 of 96.7%, 99.8% and 100.0% in text retrieval, respectively; and R@1, R@5 and R@10 of 85.9%, 98.3% and 99.0% in image retrieval, respectively. On the MSCOCO dataset, the model achieved R@1, R@5 and R@10 of 79.8%, 95.3% and 98.2% in text retrieval, respectively; and R@1, R@5 and R@10 of 63.4%, 86.1% and 92.3% in image retrieval, respectively. These results show that our multi-granularity alignment method performs well in cross-modal retrieval tasks. In the visual question answering (VQA) task, our model achieves 77.23% accuracy on the test-dev dataset, which is 3.31% higher than the ALBEF model; and 77.48% accuracy on the test-std dataset, which is 2.78% higher than the ALBEF model. These results show the excellent performance of our model in understanding and answering visual questions. In the natural language visual reasoning task, our model achieves 83.91% accuracy on the dev dataset, which is 3.91% higher than the base model; and 83.76% accuracy on the Test-P dataset, which is 3.26% higher than the base model. This shows that our model performs well in understanding and reasoning about natural language and visual relationships. In the visual entailment task, our method achieves significant improvement on the validation dataset, which is 2.59% higher than the base model; and 1.62% higher on the test dataset. These results verify the effectiveness and robustness of our method in accurately classifying the relationship between images and text. Figure 4-5 shown.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A general vision-language pre-training framework for multi-granular cross-modal alignment, characterized in that: The following steps are involved: Step S1, using image-text pairs as training data to input into an image encoder and a text encoder respectively to obtain coarse-grained global features and fine-grained local features; Step S2: performing multi-granularity cross-modal alignment on the global features and local features obtained in step S1, and using the aligned image and text global features for cross-modal pre-training; Step S3, the multi-granularity cross-modal alignment in step S2 consists of a text-aware image block selection module and a cross-granularity alignment module; Step S4: The text-aware image block selection module in step S3 selects the image block with the highest correlation according to the correlation between the text words or phrases and the image blocks to form the corresponding area or target object, thereby achieving more adequate fine-grained alignment; The cross-granularity alignment module in step S5 and step S3 further integrates multi-granularity information while realizing fine-grained information mining by cross-aligning text local features with image global features and image local features with text global features respectively; Step S6, the text-aware image block selection module in step S4 and the cross-granularity alignment module in step S5 jointly construct a multi-granularity cross-modal pre-training target; The cross-modal pre-training objectives in step S7 and step S2 include the multi-granularity cross-modal pre-training objectives constructed in step S6 as well as the general image-text contrast loss, image-text matching loss and masked text modeling loss.

2. A general framework for visual-language pre-training for multi-granularity cross-modal alignment according to claim 1, characterized in that: In step S1, image-text pairs are used as training data D=(V i ,T i ), where each image sample V i is divided into image blocks {p1,p2,p3,...,p N } and the image block is cls Input the image encoder to get the image global feature V g and local image features V l ={v1,v2,v3,...,v N }, and each text sample is segmented into {w1,w2,w3,...,w N } and with w cls Input the text encoder to get the global feature T of the text g and local image features T l ={t1,t2,t3,...,t N }.

3. The general framework for visual-language pre-training for achieving multi-granularity cross-modal alignment according to claim 1, characterized in that: Implement the multi-granularity cross-modal alignment in step S2 on the global features and local features obtained in step S1; The multi-granularity cross-modal alignment in step S2 is composed of a text-aware image block selection module and a cross-granularity alignment module.

4. The general framework for visual-language pre-training for achieving multi-granularity cross-modal alignment according to claim 1, characterized in that: The text-aware image block selection module in step S4 selects the image block with the highest correlation according to the correlation between the text words or phrases and the image blocks to form the corresponding area or target object, and screens out the image local features and text local features corresponding to the image blocks and words rich in important information in the image and text respectively based on the self-attention mechanism. and Represented by V, and together with the image and text global features, constitute the output of the text-aware image patch selection module. s and T s On this basis, the cross-attention mechanism is further used to select the local image features that are most relevant to the local text features. Its essence is to select the local features of the image that are most correlated with the local features of the text through the attention feature map, so as to construct the area or target object corresponding to the local features of the text to achieve more sufficient fine-grained alignment:

5. The general framework for visual-language pre-training for achieving multi-granularity cross-modal alignment according to claim 1, characterized in that: The cross-granularity alignment module in step S5 respectively cross-aligns the local features of the text with the global features of the image and the local features of the image with the global features of the text, and the global features of the text according to the global features of the image and text obtained in step S1. g and T g And the local features selected in step S4 and Compute the similarity of cross-granular alignment: Where S I-W represents the similarity between each image sample and the words in each group of text samples, S T-P Represents the similarity between each text sample and the image block in each group of image samples, so as to calculate the cross-granularity similarity through the inner product:

6. The general vision-language pre-training framework for implementing multi-granularity cross-modal alignment according to claim 1, characterized in that: The multi-granularity cross-modal pre-training target constructed in step S6 further mines fine-grained information and fuses multi-granularity information. The multi-granularity similarity between image and text is expressed as: s mlga (I,T)=αs global (I,T)+(1-α)s local (I,T) The multi-granularity cross-modal pre-training objective is expressed as: Where (I,T) represents the matched image-text pair, (I,T k ) and (I k ,T) represent the unmatched image-text pairs, η represents the threshold, and the default value is set to 0.

2.

7. The general framework for visual-language pre-training for achieving multi-granularity cross-modal alignment according to claim 1, characterized in that: The cross-modal pre-training target constructed in step S7 includes, in addition to the multi-granular cross-modal pre-training target, a general image-text contrast loss, an image-text matching loss, and a masked text modeling loss. g and T g , the image-text similarity s(I,T) can be expressed as V g (T g ) T , the text-image similarity and image-text similarity are expressed as: Where τ represents the temperature parameter, and the image-text contrast loss is expressed by the cross entropy function: The global features of the image and the global features of the text are used as input to obtain cross-modal representation through the cross-modal fusion encoder to determine whether the image and the text match. The image-text matching loss is expressed as: The masked text modeling loss is expressed as: in Represents the probability of the model's prediction for each mask tag, and the final complete loss function is expressed as:

Images