Comparative language image model training method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of computers, and provides a training method and device for a contrast language image model, electronic equipment and a medium. The method comprises the following steps: acquiring a text sample and an image sample; obtaining a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample according to the first contrast language image model; obtaining a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample according to the second contrast language image model; constructing a first loss function based on the first text sample feature and the second text sample feature, constructing a second loss function based on the first image sample feature and the second image sample feature, and constructing a third loss function based on the second text sample feature and the second image sample feature; and training the second contrast language image model to obtain a trained second contrast language image model. The problem that the application potential of a contrast language image model is limited is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, device, electronic device and medium for contrasting language image models. Background Art

[0002] The Contrastive Language-Image Pretraining (CLIP) model effectively integrates the semantic information of images and text through contrastive learning, becoming a leading visual language pretraining model. This feature of CLIP enables it to demonstrate excellent performance in multiple multimodal tasks such as image recognition, visual question answering, and image description generation. Despite this, large CLIP models are difficult to deploy in resource-constrained environments due to their high resource consumption. Although small CLIP models have lower resource consumption, their performance often cannot meet the needs of actual applications, which limits the application of contrastive language image models in actual scenarios. Summary of the invention

[0003] In view of this, the present application provides a training method, device, electronic device and medium for a contrastive language image model to solve the problem of limited application potential of the contrastive language image model in the prior art.

[0004] In a first aspect of the present application, a training method for a contrastive language image model is provided, comprising: obtaining a text sample and an image sample, the text sample and the image sample corresponding to each other; processing the text sample and the image sample respectively according to a first contrastive language image model to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample; processing the text sample and the image sample respectively according to a second contrastive language image model to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample; constructing a first loss function based on the first text sample feature and the second text sample feature, constructing a second loss function based on the first image sample feature and the second image sample feature, and constructing a third loss function based on the second text sample feature and the second image sample feature; training the second contrastive language image model based on the first loss function, the second loss function and the third loss function to obtain a trained second contrastive language image model.

[0005] In a second aspect of the present application, a training device for a contrastive language image model is provided, comprising: an acquisition module, configured to acquire text samples and image samples, the text samples and the image samples corresponding to each other; a first processing module, configured to process the text samples and the image samples respectively according to the first contrastive language image model, and obtain first text sample features corresponding to the text samples and first image sample features corresponding to the image samples; a second processing module, configured to process the text samples and the image samples respectively according to the second contrastive language image model, and obtain second text sample features corresponding to the text samples and second image sample features corresponding to the image samples; a loss construction module, configured to construct a first loss function based on the first text sample features and the second text sample features, construct a second loss function based on the first image sample features and the second image sample features, and construct a third loss function based on the second text sample features and the second image sample features; a training module, configured to train the second contrastive language image model based on the first loss function, the second loss function and the third loss function, and obtain a trained second contrastive language image model.

[0006] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of the present application, a medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] At least one of the above technical solutions adopted in this application can achieve the following beneficial effects:

[0009] By acquiring text samples and image samples, the text samples and the image samples correspond to each other; according to the first contrastive language image model, the text samples and the image samples are processed respectively to obtain first text sample features corresponding to the text samples and first image sample features corresponding to the image samples; according to the second contrastive language image model, the text samples and the image samples are processed respectively to obtain second text sample features corresponding to the text samples and second image sample features corresponding to the image samples; a first loss function is constructed based on the first text sample features and the second text sample features, a second loss function is constructed based on the first image sample features and the second image sample features, and a third loss function is constructed based on the second text sample features and the second image sample features; based on the first loss function, the second loss function and the third loss function, the second contrastive language image model is trained to obtain a trained second contrastive language image model. In this way, multiple loss functions can be introduced to measure the feature alignment between the first comparative language image model and the second comparative language image model, and the text and image feature representation of the second comparative language image model can be optimized so that it can be close to or match the performance of the first comparative language image model. Based on this, the second comparative language image model can be helped to learn the knowledge of the first comparative language image model through contrastive learning and knowledge distillation methods, thereby broadening its potential in practical applications. This can avoid the problem that the first comparative language image model is difficult to deploy in an environment due to its large resource consumption, and the second comparative language image model has low resource consumption but its performance cannot meet the needs of practical applications. This allows the second comparative language image model to maintain good performance even when resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 It is a flowchart of a training method for a comparative language image model provided in an embodiment of the present application;

[0012] Figure 2 It is a flowchart of another method for training a comparative language image model provided in an embodiment of the present application;

[0013] Figure 3 It is a flowchart of another method for training a comparative language image model provided in an embodiment of the present application;

[0014] Figure 4It is a structural schematic diagram of a training device for contrasting language image models provided in an embodiment of the present application;

[0015] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0017] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0018] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0019] The CLIP model can be used to process the relationship between images and texts, and can embed images and texts into the same vector space to achieve multimodal understanding. However, the CLIP model is limited by resources and environment in actual application.

[0020] To overcome the challenges of CLIP model deployment in resource-constrained environments and its performance limitations when encoding textual information, multiple knowledge distillation (KD) strategies have been explored. TinyCLIP adopts affinity imitation and weight inheritance to leverage the strengths of large models to enhance the capabilities of small models. CLIP-TD aims to improve the performance of models by distilling the knowledge of CLIP's vision and language branches into existing image-language task architectures. PromptKD significantly improves the performance of small models through prompt-based knowledge distillation using unlabeled domain data. Although these methods are all committed to distilling the knowledge of the teacher network based on the results of text-image feature fusion, they often ignore the deep knowledge implied in the feature fusion process. This oversight limits the efficiency of the student network in absorbing knowledge from the teacher network. In addition, since the textual information in the dataset is usually simple and incomplete descriptions, CLIP faces performance bottlenecks when encoding textual information, especially in tasks such as color recognition and quantity estimation.

[0021] In order to solve these problems, this scheme, namely the training method of contrastive language image model, is proposed. This scheme not only focuses on refining the feature fusion results of large models, but also strives to explore the knowledge contained in the feature fusion process. In this way, the student network can absorb the knowledge of the teacher network more comprehensively. In addition, by leveraging the rich corpus knowledge of the large language model (LLM), the text encoder of CLIP can encode more fine-grained text features. This method uses knowledge distillation to enable the text encoder of CLIP to more accurately capture complex concepts in the text, thereby improving the performance of the model in multimodal tasks and solving the problem of limited practical application of the model. It should be understood that the contrastive language image model referred to in this scheme can represent the CLIP model.

[0022] A method and device for training a contrastive language image model according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 is a flow chart of a method for training a comparative language image model provided in an embodiment of the present application. Figure 1 As shown, the training method of the contrastive language image model includes:

[0024] S101, obtaining text samples and image samples, the text samples and the image samples corresponding to each other.

[0025] Specifically, the text samples and image samples may represent text sample data and image sample data used to train the model. The text samples may be expressed as sentences, phrases or descriptions, and the image samples may be expressed as photos, illustrations, etc. The text samples and image samples in the training process are usually corresponding.

[0026] S102: Process the text sample and the image sample respectively according to the first contrasting language image model to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample.

[0027] Specifically, the first contrastive language image model can represent the large CLIP model as a teacher network for knowledge distillation, which is used to perform knowledge distillation on the small CLIP model, so that the small CLIP model can fully learn the knowledge of the large CLIP model, which can further expand the potential of small CLIP in practical applications.

[0028] S103, processing the text sample and the image sample respectively according to the second contrasting language image model to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample.

[0029] Specifically, the second contrastive language image model can represent the small CLIP model as a student network for knowledge distillation, that is, it can be used to learn the knowledge of the large CLIP model and improve its capabilities in practical applications.

[0030] S104, constructing a first loss function based on the first text sample feature and the second text sample feature, constructing a second loss function based on the first image sample feature and the second image sample feature, and constructing a third loss function based on the second text sample feature and the second image sample feature.

[0031] S105 , training the second contrastive language image model based on the first loss function, the second loss function, and the third loss function to obtain a trained second contrastive language image model.

[0032] In some embodiments, according to the second contrastive language image model, the text sample and the image sample are processed respectively to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample, including: using the first text encoder to process the text sample to obtain a third text sample feature; expanding the third text sample feature to obtain a fourth text sample feature, and using a multilayer perceptron to process the fourth text sample feature to obtain a second text sample feature; using the first image encoder to process the image sample to obtain a third image sample feature, and cross-modally learning the third image sample feature based on the first text sample feature and the second text sample feature to obtain a second image sample feature; wherein the second contrastive language image model includes a first text encoder, a multilayer perceptron, and a first image encoder.

[0033] Specifically, the text sample can be preliminarily processed by the first text encoder to obtain the third text sample feature, and then the third sample text feature can be expanded to enhance the richness of the text feature and generate a more complex and fine-grained fourth text sample feature. The fourth text sample feature can then be input into a multilayer perceptron (Multilayer Perceptron, MLP) for further nonlinear transformation to obtain a more optimized text representation, namely the second text sample feature. It should be noted that the multilayer perceptron used in the embodiment of the present application can represent a feedforward network for introducing nonlinearity. The multilayer perceptron used in multiple embodiments of the present application can be the same. Of course, in specific applications, adaptive adjustments can be made based on specific task requirements, which is not limited here.

[0034] Similarly, the first image encoder can be used to process the image samples, extract preliminary image features, obtain third image sample features, and further perform cross-modal learning on the third image sample features. At this time, the third image sample features can be cross-modally learned based on the first text sample features and the second text sample features, and the second image sample features can be generated based on the third image sample features through guidance of the text features.

[0035] According to the technical solution provided in the above embodiment, multimodal contrastive learning can be used to first extract basic features through the first text encoder and the first image encoder, and then expand and map the third text sample features and use cross-modal learning to optimize the third image sample features. This can achieve accurate alignment of image and text features, so that the second contrastive language image model can better understand the semantic relationship between image and text.

[0036] In some embodiments, the third text sample feature is expanded to obtain a fourth text sample feature, including: processing the third text sample feature through a multilayer perceptron to obtain a fifth text sample feature; aligning the fifth text sample feature with the sixth text sample feature according to a preset alignment rule to obtain a new fifth text sample feature, wherein the sixth text feature represents a feature obtained by processing the text sample using a large language model; and adding the new fifth text sample feature to the third text sample feature to obtain a fourth text sample feature.

[0037] Specifically, during the model training process, after obtaining the third text sample feature, it can be expanded to capture more complex semantic information. The third text sample feature can represent the feature representation obtained by the first text encoder extracting the text sample, which is a basic understanding of the text. Therefore, during the model training process, the third text sample feature can be expanded through semantic enrichment to enhance its expressiveness.

[0038] Furthermore, the third text sample feature can be processed using a multilayer perceptron to obtain the fifth text sample feature, so that the third text sample feature can be nonlinearly transformed to capture more complex details and make its feature expression ability stronger.

[0039] Furthermore, the fifth text sample feature and the sixth text sample feature can be aligned using a preset alignment rule, wherein the sixth text sample feature can be obtained by expanding the text sample according to the large language model to capture more semantic information about the text and help improve the richness of the feature representation. The preset alignment function can be a divergence alignment (Kullback-Leibler Divergence, KL), and KL divergence alignment can represent a tool for measuring the difference between two probability distributions. Aligning the fifth text sample feature with the sixth text sample feature can make the distribution of the fifth text sample feature as close as possible to the distribution of the sixth text sample feature, and then make the fifth text sample feature obtained by multi-layer perceptron processing close to the expanded and richer sixth text sample feature, and obtain a new fifth text sample feature, so that the model can better capture the deep semantic information in the text.

[0040] Furthermore, the new fifth text sample feature can be added to the third text sample feature, and the new fifth text sample feature and the third text sample can be linearly weighted through hyperparameters to obtain the fourth text sample feature. In this way, the new fifth text sample feature obtained after expansion can be fused with the preliminary third text sample feature to obtain a new and more optimized feature representation, namely the fourth text sample feature, which not only retains the key information in the preliminary third text sample feature, but also includes the deep semantic details introduced by expansion and alignment.

[0041] According to the technical solution provided in the above embodiment, the third text sample feature is processed by a multi-layer perceptron to obtain the fifth text sample feature, and the fifth text sample feature is aligned with the sixth text sample feature, so that the distribution of the fifth text sample feature is closer to the expanded feature, and finally the new fifth text sample feature is added to the third text sample feature to obtain a richer and more accurate feature representation, which can fully enhance the expressive power of the text feature and enable the face model to better capture and process the complex semantics in the text.

[0042] In some embodiments, the second contrastive language image model also includes a cross-modal attention module, which performs cross-modal learning on the third image sample features based on the first text sample features and the second text sample features to obtain second image sample features, including aligning the third image sample features with the first image sample features to obtain new third image sample features; using a multi-layer perceptron to process the new third image sample features to obtain fourth image sample features; and using the cross-modal attention module to process the first text sample features, the second text sample features and the fourth image sample features to obtain second image sample features.

[0043] Specifically, the cross-module attention module can represent a module that processes image features through an attention mechanism to achieve cross-modal learning.

[0044] Before performing cross-modal learning, the third image sample features may be processed first, that is, the third image sample features are aligned with the first image sample features by KL divergence, and the third image sample features are adjusted to be similar to the first image sample features in the large teacher network, i.e., the first contrastive language image model, so as to learn the task processing method of the first contrastive image model. In the process of aligning the third image sample features with the first image sample features, the third image sample features may be aligned with the first image sample features through a loss function to obtain a new third image sample feature, and the loss function may adopt contrast loss, which is not limited here.

[0045] Furthermore, after obtaining the new third image sample features, a multilayer perceptron can be used to further perform nonlinear transformation on them to extract and enhance the expressive power of the features, thereby obtaining rich fourth image sample features.

[0046] Furthermore, the fourth image sample feature, the first text sample feature, and the second text sample feature can be input into the cross-modal attention module for processing. It should be understood that the cross-modal attention module can be represented as a Qformer structure, namely, a Querying Transformer, which performs visual-language alignment between the frozen visual model and the large language model. Furthermore, the first text sample feature can be used as a query input, and the second text sample feature can be input at the same time. In this way, Qformer can better capture the correlation between text and image. In this way, the attention mechanism can be used to allow the model to dynamically select the image features that are most relevant to the text features, thereby extracting an image representation with more semantic understanding and obtaining the second image sample feature.

[0047] According to the technical solution provided in the above embodiment, the third image sample features are processed by KL divergence alignment to make them closer to the feature representation of the teacher network, and the new third image sample features obtained after the alignment are processed using a multilayer perceptron to enhance their expressiveness. Then, the first text sample features are combined with the second text sample features by using a cross-modal attention module, and the image features are dynamically selected and processed to extract the second image sample features. This effectively combines the text and image features, and utilizes the cross-modal learning mechanism to improve the performance of the second comparative language image model in multimodal tasks.

[0048] In some embodiments, text samples and image samples are processed according to a first contrastive language image model to obtain first text sample features corresponding to the text samples and first image sample features corresponding to the image samples, respectively, including: processing the text samples using a second text encoder to obtain first text sample features; processing the image samples using a second image encoder to obtain first image sample features; wherein the first contrastive language image model includes a second text encoder and a second image encoder.

[0049] Specifically, when training the second contrastive language image model (i.e., the student network), the processing effect of the first contrastive language image model (i.e., the teacher network) can be understood. The first contrastive language image model may include a second text encoder and a second image encoder, wherein the second text encoder may process text samples through a series of neural network layers (e.g., embedding layers, LSTM, Transformer, etc.) to capture semantic information in the text samples for the student network to learn. The second image encoder may convert image samples into feature representations in the form of vectors. The second image encoder may include convolutional layers, pooling layers, and other feature extraction components, which are not limited here, and can extract spatial information set features from images.

[0050] According to the technical solution provided in the above embodiment, extracting the first text sample features through the second text encoder and extracting the first image sample features through the second image encoder can provide a basis for subsequent student network learning and promote the student network's understanding of the relationship between text and images.

[0051] In some embodiments, based on the first loss function, the second loss function and the third loss function, the second contrastive language image model is trained to obtain the trained second contrastive language image model, including: based on the first loss function and the third loss function, the first text encoder is trained to obtain a new first text encoder; based on the second loss function and the third loss function, the first image encoder is trained to obtain a new first image encoder, and the trained second contrastive language image model includes a new first text encoder and a new first image encoder.

[0052] Specifically, when training the second contrastive language image model, the training may be performed using a loss function, such as a mean square error loss function, to measure the accuracy of the corresponding feature alignment.

[0053] It should be noted that the first loss function can be expressed as the alignment loss between the first text sample feature and the second text sample feature, which can make the two features as close as possible in space; the second loss function can be understood as the alignment loss between the first image sample feature and the second image sample feature, which can also make the two features as close as possible in the feature space; the third loss function can be expressed as the contrast loss between the second sample text feature and the second image sample feature. By minimizing the contrast loss, the image features generated in the student network can be as close to the text features as possible.

[0054] Then, the first text encoder is trained based on the first loss function and the third loss function, and the parameters of the first text encoder are updated through the back propagation algorithm to obtain an optimized new first text encoder. Similarly, the parameters of the first image encoder are updated based on the second loss function and the third loss function to obtain an optimized new first image encoder. In this way, the new first text encoder can more accurately extract text features, and the new first image encoder can better extract image features, so that the performance of the student network is sufficient to meet practical application needs.

[0055] In some embodiments, after training the second contrastive language image model to obtain the trained second contrastive language image model, it also includes: acquiring text and an image, the text corresponding to the image; processing the text using a new first text encoder to obtain a first text feature; processing the first text feature through a multilayer perceptron to obtain a second text feature; adding the first text feature to the second text feature to obtain an addition result, and processing the addition result through a multilayer perceptron to obtain a third text feature; processing the image using a new first image encoder to obtain an initial image feature; processing the initial image feature and the third text feature using a cross-modal attention module to obtain image features output by the trained second contrastive language image model.

[0056] Figure 2 is a flow chart of another method for training a contrastive language image model provided in an embodiment of the present application, which can represent the task processing process of the trained second contrastive language image model. Figure 2 This embodiment is described.

[0057] Specifically, after obtaining the trained second contrastive language image model, the text and image can be processed and image features that conform to the text description can be output.

[0058] First, the text and image to be processed can be obtained, and the text can be processed using the new first text encoder to better extract text features and obtain text features (i.e., the first text features in the previous text). The first text features are processed by a multi-layer perceptron for nonlinear transformation to enhance the expressiveness of the text features and obtain text features (i.e., the second text features). The original first text features are added and fused with the second text features processed by the multi-layer perceptron, retaining information at different levels, and the addition result is processed by MLP again to further extract richer features and obtain text features (i.e., the third text features). The third text features will be used for cross-modal processing with image features.

[0059] At the same time, the new first image encoder is used to extract features from the image to obtain initial image features, which can represent visual features of the image.

[0060] Furthermore, a cross-modal attention module (i.e., Qformer) can be used to fuse text and image features, that is, the third text feature is used as a query for the image feature to help the model better understand the semantic information of the image. Qformer can find the corresponding semantic clues in the initial image features based on the query of the third text feature and then adjust or optimize it, and finally output the target image feature.

[0061] It should be understood that these target image features can be used for various multimodal tasks, such as image-text matching, image classification or other tasks, which are not limited here.

[0062] According to the technical solution provided in the above embodiment, the trained second contrastive language model can fully learn the features of text and image and better capture the relationship between text and image, thereby improving the performance of the model in multimodal tasks.

[0063] Figure 3 This is a flow chart of another method for training a comparative language image model provided in an embodiment of the present application. Figure 3 The embodiments of the present application are further described.

[0064] Figure 3 It can be understood as the training process of the student network, that is, how to train to obtain the trained second contrast image model.

[0065] First, the text sample and the image sample can be input into the student network and the teacher network respectively. In the teacher network, the text sample is processed by the second text encoder to obtain the first text sample feature, and the image sample is processed by the second image encoder to obtain the first image sample feature, which is stored for subsequent use.

[0066] In the student network, the text sample is processed by the first text encoder to obtain the third text sample feature, the third text sample feature is processed by a multi-layer perceptron to obtain the fifth text sample feature, and the text sample is processed by a large language model to obtain the sixth text sample feature, the fifth text sample feature is aligned to the sixth text sample feature to obtain a new fifth text sample feature, and the new fifth text sample feature is added and fused with the third text sample feature to obtain the fourth text sample feature, and the fourth text sample feature is further linearly transformed by a multi-layer perceptron to obtain the second text sample feature.

[0067] At the same time, the image samples are processed by the first image encoder to obtain the third image sample features, and are processed by the multilayer perceptron to obtain the fourth image sample features. The first text sample features obtained in the teacher network and the second sample text features obtained in the student network are input into Qformer to process the fourth image sample features to obtain the second image sample features. The alignment loss and contrast loss are calculated and the loss is minimized. The student network can be trained to obtain the trained student network, that is, the trained second contrast language image model.

[0068] In this way, the knowledge of the teacher network can be transferred to the student network, making the feature representation of the student network close to that of the teacher network. Through the above steps, the student network can learn the feature representation of the teacher network while maintaining consistency in the feature space, thereby improving the performance and generalization ability of the model. In the framework of the comparative language-image pre-training model algorithm, by refining the knowledge of the large model in the feature fusion process and combining the knowledge distillation of LLM, the deployment efficiency of the small CLIP model in resource-constrained environments and its performance in encoding text information can be significantly improved, further broadening its potential in practical applications.

[0069] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0070] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0071] Figure 4 Schematic diagram of a training device for comparing language images provided in an embodiment of the present application. Figure 4As shown, the training device for comparing language images includes:

[0072] An acquisition module 401 is configured to acquire text samples and image samples, the text samples and the image samples corresponding to each other;

[0073] A first processing module 402 is configured to process the text sample and the image sample according to the first contrastive language image model, respectively, to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample;

[0074] The second processing module 403 is configured to process the text sample and the image sample according to the second contrasting language image model, respectively, to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample;

[0075] A loss construction module 404 is configured to construct a first loss function based on the first text sample feature and the second text sample feature, construct a second loss function based on the first image sample feature and the second image sample feature, and construct a third loss function based on the second text sample feature and the second image sample feature;

[0076] The training module 405 is configured to train the second contrastive language image model based on the first loss function, the second loss function and the third loss function to obtain a trained second contrastive language image model.

[0077] In some embodiments, the second processing module 403 is specifically used to process the text sample using the first text encoder to obtain a third text sample feature; expand the third text sample feature to obtain a fourth text sample feature, and process the fourth text sample feature using a multilayer perceptron to obtain a second text sample feature; process the image sample using the first image encoder to obtain a third image sample feature, and based on the first text sample feature and the second text sample feature, perform cross-modal learning on the third image sample feature to obtain the second image sample feature; wherein the second comparative language image model includes a first text encoder, a multilayer perceptron, and a first image encoder.

[0078] In some embodiments, the second processing module 403 is specifically used to process the third text sample feature through a multi-layer perceptron to obtain a fifth text sample feature; align the fifth text sample feature with the sixth text sample feature according to a preset alignment rule to obtain a new fifth text sample feature, where the sixth text feature represents the feature obtained by processing the text sample using a large language model; add the new fifth text sample feature to the third text sample feature to obtain a fourth text sample feature.

[0079] In some embodiments, the second processing module 403 is specifically used to align the third image sample feature with the first image sample feature to obtain a new third image sample feature; use a multi-layer perceptron to process the new third image sample feature to obtain a fourth image sample feature; use a cross-modal attention module to process the first text sample feature, the second text sample feature and the fourth image sample feature to obtain a second image sample feature.

[0080] In some embodiments, the first processing module 402 is specifically used to process the text sample using the second text encoder to obtain the first text sample features; and to process the image sample using the second image encoder to obtain the first image sample features; wherein the first comparative language image model includes the second text encoder and the second image encoder.

[0081] In some embodiments, the training module 405 is specifically used to train the first text encoder based on the first loss function and the third loss function to obtain a new first text encoder; based on the second loss function and the third loss function, train the first image encoder to obtain a new first image encoder, and the trained second comparative language image model includes the new first text encoder and the new first image encoder.

[0082] In some embodiments, the training module 405 is specifically used to obtain text and images, the text and the image correspond to each other; use a new first text encoder to process the text to obtain a first text feature; process the first text feature through a multilayer perceptron to obtain a second text feature; add the first text feature to the second text feature to obtain an addition result, and process the addition result through a multilayer perceptron to obtain a third text feature; use a new first image encoder to process the image to obtain an initial image feature; use a cross-modal attention module to process the initial image feature and the third text feature to obtain a target image feature output by the trained second comparative language image model.

[0083] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present application. Figure 5As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0085] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0086] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0087] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0088] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0089] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0090] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for training a contrastive language image model, characterized in that: include: Acquire a text sample and an image sample, wherein the text sample corresponds to the image sample; According to the first contrasting language image model, the text sample and the image sample are processed respectively to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample; According to the second contrasting language image model, the text sample and the image sample are processed respectively to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample; Constructing a first loss function based on the first text sample feature and the second text sample feature, constructing a second loss function based on the first image sample feature and the second image sample feature, and constructing a third loss function based on the second text sample feature and the second image sample feature; Based on the first loss function, the second loss function and the third loss function, the second contrastive language image model is trained to obtain a trained second contrastive language image model.

2. The method according to claim 1, characterized in that The processing of the text sample and the image sample according to the second contrasting language image model to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample includes: Processing the text sample using the first text encoder to obtain a third text sample feature; Expanding the third text sample feature to obtain a fourth text sample feature, and processing the fourth text sample feature using a multilayer perceptron to obtain the second text sample feature; Processing the image sample using a first image encoder to obtain a third image sample feature, and performing cross-modal learning on the third image sample feature based on the first text sample feature and the second text sample feature to obtain the second image sample feature; The second contrastive language image model includes the first text encoder, the multilayer perceptron and the first image encoder.

3. The method according to claim 2, characterized in that The step of expanding the third text sample feature to obtain a fourth text sample feature includes: Processing the third text sample feature through the multi-layer perceptron to obtain a fifth text sample feature; According to a preset alignment rule, the fifth text sample feature is aligned with the sixth text sample feature to obtain a new fifth text sample feature, wherein the sixth text feature represents a feature obtained by processing the text sample using a large language model; The new fifth text sample feature is added to the third text sample feature to obtain the fourth text sample feature.

4. The method according to claim 2, characterized in that: The second contrastive language image model further includes a cross-modal attention module, and the cross-modal learning of the third image sample feature based on the first text sample feature and the second text sample feature to obtain the second image sample feature includes: Aligning the third image sample feature with the first image sample feature to obtain a new third image sample feature; Processing the new third image sample feature using the multilayer perceptron to obtain a fourth image sample feature; The first text sample feature, the second text sample feature, and the fourth image sample feature are processed using the cross-modal attention module to obtain the second image sample feature.

5. The method according to claim 1, characterized in that The processing of the text sample and the image sample according to the first contrasting language image model to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample respectively includes: Processing the text sample using a second text encoder to obtain the first text sample feature; Processing the image sample using the second image encoder to obtain features of the first image sample; The first contrastive language image model includes the second text encoder and the second image encoder.

6. The method according to claim 2, characterized in that The step of training the second contrastive language image model based on the first loss function, the second loss function, and the third loss function to obtain the trained second contrastive language image model includes: Based on the first loss function and the third loss function, training the first text encoder to obtain a new first text encoder; Based on the second loss function and the third loss function, the first image encoder is trained to obtain a new first image encoder, and the trained second contrastive language image model includes the new first text encoder and the new first image encoder.

7. The method according to claim 6, characterized in that: After training the second contrasting language image model to obtain the trained second contrasting language image model, the method further includes: Acquire text and an image, wherein the text and the image correspond to each other; Processing the text using the new first text encoder to obtain a first text feature; Processing the first text feature through the multi-layer perceptron to obtain a second text feature; Adding the first text feature and the second text feature to obtain an addition result, and processing the addition result through the multilayer perceptron to obtain a third text feature; Processing the image using the new first image encoder to obtain initial image features; The initial image features and the third text features are processed using the cross-modal attention module to obtain target image features output by the trained second contrastive language image model.

8. A training device for contrasting language images, characterized in that: include: An acquisition module, configured to acquire a text sample and an image sample, wherein the text sample corresponds to the image sample; A first processing module is configured to process the text sample and the image sample respectively according to a first contrasting language image model to obtain a first text sample feature corresponding to the text sample and a first image sample feature corresponding to the image sample; A second processing module is configured to process the text sample and the image sample respectively according to a second contrasting language image model to obtain a second text sample feature corresponding to the text sample and a second image sample feature corresponding to the image sample; A loss construction module, configured to construct a first loss function based on the first text sample feature and the second text sample feature, construct a second loss function based on the first image sample feature and the second image sample feature, and construct a third loss function based on the second text sample feature and the second image sample feature; The training module is configured to train the second contrastive language image model based on the first loss function, the second loss function and the third loss function to obtain a trained second contrastive language image model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images