Image-text processing method and system based on lightweight CLIP model

Abstract

The invention relates to an image-text processing method and system based on a lightweight CLIP model, and the method comprises the steps: obtaining to-be-processed image-text data, carrying out the feature extraction of the image-text data through a lightweight CLIP model, and carrying out the image text retrieval or image text classification based on the extracted features; wherein the lightweight CLIP model is an improved CLIP model obtained by replacing a Pre-LN block in an original CLIP model with an SAS-P block containing an attention mechanism; the training of the improved model comprises the steps that the improved CLIP model serves as a student model, an original CLIP model serves as a teacher model, model training is carried out, comparison loss, pairing loss and multi-stage knowledge distillation loss are adopted in the model training to update and improve CLIP model parameters, and the system is used for achieving the method. Compared with the prior art, the lightweight CLIP model for image-text processing provided by the invention has the advantages that the scale of the CLIP model is reduced, and the model calculation and training cost is reduced, so that efficient image-text processing is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image and text processing method and system based on a lightweight CLIP model. Background Art

[0002] Existing image-text processing methods mainly follow the general pattern of image encoding and text decoding. Specifically, these methods usually use an image encoder to extract visual features and then use a text decoder to generate the corresponding text description. However, due to the difference between visual features and semantic features, the training of these models usually requires a lot of computing resources. Taking the Contrastive Language Image Pretraining (CLIP) model as an example, the high computing and storage costs required to train such CLIP models have hindered their further popularization. For example, MobileCLIP is trained on a 256×A100 GPU with a global batch size of 65536, and the corresponding dataset DataCompDR-1B requires 140TB of local storage space. In addition, the large parameter scale (for example, the image encoder parameters of the CLIP-B / 16 model are as high as 86.2M and the text encoder parameters are as high as 63.4M) also leads to increased inference latency, which poses a challenge to deploying the model on devices with limited computing resources. These shortcomings make it difficult for ordinary users with insufficient computing resources and datasets to participate in the training and deployment of such large-scale models. In fact, the GPU memory of consumer-grade computers usually does not exceed 24GB (for example, Nvidia RTX3090), while the storage capacity may be less than 1TB. In the existing technology, model performance is improved by fine-tuning the original CLIP model. For example, Chinese patent application "CN118097317A" provides a remote sensing image classification method based on large-model secondary distillation. It discloses a method for fine-tuning the CLIP model based on the LoRA method, and then optimizes model performance through secondary distillation. Although this improves the processing efficiency of the CLIP model and thus the efficiency of remote sensing image classification, it still relies on fine-tuning the original CLIP model, and still cannot solve the problem of "the high computational and storage costs required to train such CLIP models" in the existing technology. In addition, the secondary distillation period requires high-quality influence labels. When the data label quality is inconsistent, the model's generalization ability cannot be guaranteed, which limits the image classification capability.

[0003] Therefore, providing a method that can reduce the scale of the CLIP model, reduce model calculation and training costs to achieve efficient image and text processing is a technical problem that needs to be solved. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a graphic and text processing method and system based on a lightweight CLIP model. The original CLIP model is structurally simplified and then multi-stage knowledge distillation is performed. The parameters of the improved CLIP model are updated by calculating the pairing loss, contrast loss and multi-stage knowledge distillation loss during the training process.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to a first aspect of the present invention, a method for image and text processing based on a lightweight CLIP model is provided, the method comprising:

[0007] Acquire image and text data to be processed, perform feature extraction on the image and text data using a lightweight CLIP model, and perform image and text retrieval or image and text classification based on the extracted features;

[0008] The lightweight CLIP model is an improved CLIP model obtained by replacing the Pre-LN block in the original CLIP model with a SAS-P block containing an attention mechanism. The training of the improved model includes:

[0009] The improved CLIP model is used as a student model, and the original CLIP model is used as a teacher model for model training. The model training uses contrast loss, pairing loss and multi-stage knowledge distillation loss to update the parameters of the improved CLIP model; and the multi-stage knowledge distillation includes parallel single-modal feature space distillation, contrast correlation matrix space distillation and cross-contrast space distillation.

[0010] As a preferred technical solution, the feature processing process of the SAS-P block includes:

[0011] Normalizing the processed image and text features to obtain first image and text features;

[0012] Processing the first image-text feature using an attention mechanism and a feedforward neural network respectively to obtain a second image-text feature and a third image-text feature;

[0013] Processing the first graphic feature and the second graphic feature element by element to obtain a fourth graphic feature;

[0014] The third image-text feature and the fourth image-text feature are processed element by element to obtain the output feature of the SAS-P block.

[0015] As a preferred technical solution, the training method includes:

[0016] Construct a data augmentation dataset, and randomly extract a batch of images and texts from the data augmentation dataset;

[0017] Input the pictures and texts into the original CLIP model to obtain teacher visual features and teacher text features, and input the same batch of pictures and texts into the improved CLIP model to obtain student visual features and student text features;

[0018] Performing unimodal feature space distillation based on the teacher visual features, teacher text features, student visual features, and student text features, and calculating the unimodal feature space distillation loss;

[0019] Calculating a teacher image-text correlation matrix and a student image-text correlation matrix based on the teacher visual features, the teacher text features, the student visual features, and the student text features, performing contrastive correlation matrix space distillation based on the teacher image-text correlation matrix and the student image-text correlation matrix, and calculating the contrastive correlation matrix space distillation loss;

[0020] Calculating a first bimodal feature cross-correlation matrix and a second bimodal feature cross-correlation matrix based on the teacher visual features, the teacher text features, the student visual features, and the student text features, performing cross-contrast spatial distillation based on the first bimodal feature cross-correlation matrix and the second bimodal feature cross-correlation matrix, and calculating the cross-contrast spatial distillation loss;

[0021] Calculate the multi-stage knowledge distillation loss based on the single-modal feature space distillation loss, contrastive correlation matrix space distillation loss, and cross-contrast space distillation loss;

[0022] The multi-stage knowledge distillation loss is combined with the contrast loss and pairing loss to improve the CLIP model parameter optimization.

[0023] As a preferred technical solution, the method for constructing a data augmentation dataset is: using the CC12M dataset as the initial dataset, using an image-text alignment model to generate multiple synthetic titles for each image in the CC12M dataset, and obtaining a data augmentation dataset.

[0024] As a preferred technical solution, the calculation expression of the single-modal feature space distillation loss is:

[0025]

[0026] Where |B| represents the batch size; represents the teacher’s visual features of the k-th image and text; represents the student visual features of the k-th image and text; represents the teacher text features of the k-th image; Represents the student text features of the k-th image and text.

[0027] As a preferred technical solution, the calculation expression of the contrast correlation matrix spatial distillation loss is:

[0028]

[0029] Among them, |B| represents the batch size; Sim T Represents the teacher's picture and text correlation matrix; Sim S represents the student image-text correlation matrix; KL(·) represents the KL divergence calculation.

[0030] As a preferred technical solution, the first bimodal feature cross-correlation matrix is ​​calculated, and its expression is:

[0031]

[0032] Where |B| represents the batch size; represents the student visual features of the k-th image and text; represents the teacher text feature of the kth image; τ represents the temperature parameter; represents the teacher text features of the bth image;

[0033] Calculate the second bimodal feature cross-correlation matrix, which is expressed as:

[0034]

[0035] in, Represents the student text features of the k-th image; represents the teacher’s visual features of the k-th image and text; Represents the teacher's visual features of the b-th image and text.

[0036] As a preferred technical solution, the cross-comparison spatial distillation loss calculation method is:

[0037]

[0038] in, represents the first bimodal feature cross-correlation matrix; represents the second bimodal feature cross-correlation matrix.

[0039] As a preferred technical solution, the calculation expression of the multi-stage knowledge distillation loss function is:

[0040] L KD =λ1L FD +λ2L IC +λ3L CRD ,

[0041] Among them, λ1, λ2, and λ3 are all hyperparameters; L FDrepresents the single-modal feature space distillation loss; L IC represents the cross-contrast spatial distillation loss; L CRD represents the cross-contrast spatial distillation loss.

[0042] According to a second aspect of the present invention, a graphic and text processing system based on a lightweight CLIP model is provided to implement the above method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) The present invention replaces the complex Pre-LN block in the original CLIP model with a Transformer block with a SAS-P structure that removes skip connections, value, and projection parameters. While ensuring the accuracy of extracted graphic features, it achieves faster reasoning with fewer parameters. Experimental verification shows that the improved model provided by the present invention requires 14% fewer parameters than the original model, increases reasoning speed by 3%, and achieves more efficient graphic processing.

[0045] 2) The present invention constructs a data augmentation dataset for training the improved model, uses the improved model as the student model and the original model as the teacher model, introduces multi-stage knowledge distillation in the training process, and uses the multi-stage knowledge distillation loss combined with the contrast loss and pairing loss to optimize the model parameters, so that the improved model can fit faster even when trained under a smaller batch size setting, thereby reducing the computing resources required for training. While shortening the model training process, it fully utilizes the internal correlation of image-text pairs in limited samples to perform more accurate feature processing for image-text retrieval or image-text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the lightweight CLIP model training of the present invention;

[0047] Figure 2 A comparison diagram of the Pre-LN structure and the SAS-P structure of the present invention;

[0048] Figure 3 This is an example of a graphic sample in the CC12M-SYN dataset of the present invention;

[0049] Figure 4 This is a diagram showing the experimental results of Example 2 of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0051] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0052] Example 1

[0053] In order to solve the problem in the prior art that a large amount of computing resources are needed for model training when performing image and text processing, which in turn affects the efficiency of image and text processing, two major problems must be solved when training CLIP-type models under such resource-limited circumstances. First, the number of parameters that need to be trained must be minimized while retaining the existing model knowledge as much as possible. Second, it is necessary to appropriately expand small-scale data sets and develop more effective methods to fully utilize the internal correlation of image-text pairs in limited samples. Based on this, the present invention provides an image and text processing method based on a lightweight CLIP model, which uses the lightweight CLIP model to extract features from the acquired image and text data to be processed, and performs image and text retrieval or image and text classification based on the extracted features. The lightweight CLIP model is an improved CLIP model obtained by replacing the Pre-LN block in the original CLIP model with a SAS-P block containing an attention mechanism.

[0054] In this embodiment, MobileCLIP is used as the original CLIP model, and the Pre-LN block in the model is as follows: Figure 1 The structure on the left is replaced by Figure 1 The SAS-P block with the attention mechanism shown on the right simplifies the complex processing flow of the Pre-LN block in the original model to: S1: Normalize the processed image features to obtain the first image feature; S2: Process the first image feature using the attention mechanism and the feedforward neural network to obtain the second and third image features; S3: Element-by-element processing of the first and second image features to obtain the fourth image feature; S4: Element-by-element processing of the third and fourth image features to obtain the output features of the SAS-P block. This process eliminates skip connections, value, and projection parameters, enabling the improved CLIP model to achieve faster inference than the Pre-LN block with fewer parameters.

[0055] For the above improved model, the present invention provides a training method, the process of which is as follows: Figure 2 As shown in the figure, the improved CLIP model is used as the student model and the original CLIP model is used as the teacher model for model training. The model training uses contrast loss, pairing loss and multi-stage knowledge distillation loss to optimize the parameters of the improved CLIP model; and the multi-stage knowledge distillation includes parallel single-modal feature space distillation, contrast correlation matrix space distillation and cross-contrast space distillation.

[0056] In detail, the training includes:

[0057] A1. Build a data augmentation dataset and randomly extract a batch of images and texts from the data augmentation dataset.

[0058] Most of the existing training sets are collected from the Internet, and the samples contained in them are noisy and not descriptive enough. When using small-scale datasets, the diversity and quality of data samples become more important. Adding synthetic text captions is a low-cost but effective method that can improve both diversity and quality. Therefore, in this paper, the CC12M dataset is used as the initial dataset, and the image-text alignment model is used to generate multiple synthetic captions for each image in the CC12M dataset. The effect is as follows: Figure 3 As shown in the figure, the data augmentation dataset CC12M-SYN is obtained, in which the added synthetic captions are intended to provide more text descriptions for the images, thereby improving the quality and diversity of the dataset.

[0059] In the data augmentation dataset, a picture-text sample consists of an image and a synthetic or original title. During training, for a picture-text sample, its image is enhanced by RandomResizedCrop and RandAugment, and the text is randomly selected from the original text description and synthetic text description sets.

[0060] A2. Input the images and texts into the original CLIP model to obtain the teacher's visual features and teacher's text features. Input the same batch of images and texts into the improved CLIP model to obtain the student's visual features and student's text features. Perform A3 to A5 in parallel.

[0061] A3. Perform unimodal feature space distillation based on teacher visual features, teacher text features, student visual features, and student text features, and calculate the unimodal feature space distillation loss. The calculation expression is:

[0062]

[0063] Where |B| represents the batch size; represents the teacher’s visual features of the k-th image and text; represents the student visual features of the k-th image and text; represents the teacher text features of the k-th image; Represents the student text features of the k-th image and text.

[0064] A4. Calculate the teacher image-text correlation matrix and the student image-text correlation matrix based on the teacher's visual features, teacher's text features, student's visual features, and student's text features. Perform contrastive correlation matrix spatial distillation based on the teacher's image-text correlation matrix and the student's image-text correlation matrix, and calculate the contrastive correlation matrix spatial distillation loss.

[0065] The calculation expression of the contrast correlation matrix space distillation loss is:

[0066]

[0067] Among them, |B| represents the batch size; Sim T Represents the teacher's picture and text correlation matrix; Sim S represents the student image-text correlation matrix; KL(·) represents the KL divergence calculation.

[0068] A5. Calculate the first bimodal feature cross-correlation matrix and the second bimodal feature cross-correlation matrix based on the teacher visual features, teacher text features, student visual features, and student text features, perform cross-contrast spatial distillation based on the first bimodal feature cross-correlation matrix and the second bimodal feature cross-correlation matrix, and calculate the cross-contrast spatial distillation loss.

[0069] A51. Calculate the first bimodal feature cross-correlation matrix, which is expressed as:

[0070]

[0071] Where |B| represents the batch size; represents the student visual features of the k-th image and text; represents the teacher text feature of the kth image; τ represents the temperature parameter; represents the teacher text features of the bth image;

[0072] A52. Calculate the second bimodal feature cross-correlation matrix, which is expressed as:

[0073]

[0074] in, Represents the student text features of the k-th image; represents the teacher’s visual features of the k-th image and text; Represents the teacher's visual features of the b-th image and text.

[0075] A53. Calculate the cross-contrast spatial distillation loss:

[0076]

[0077] in, represents the first bimodal feature cross-correlation matrix; represents the second bimodal feature cross-correlation matrix.

[0078] A6. Calculate the multi-stage knowledge distillation loss based on unimodal feature space distillation loss, contrastive correlation matrix space distillation loss, and cross-contrast space distillation loss.

[0079] The calculation expression of the multi-stage knowledge distillation loss function is:

[0080] L KD =λ1L FD +λ2L IC +λ3L CRD ,

[0081] Among them, λ1, λ2, and λ3 are all hyperparameters; L FD represents the single-modal feature space distillation loss; L IC represents the cross-contrast spatial distillation loss; L CRD represents the cross-contrast spatial distillation loss.

[0082] A7. Combine the multi-stage knowledge distillation loss with the contrast loss and pairing loss to improve the CLIP model parameter optimization.

[0083] During the training process, the improved CLIP model as the student model directly inherits the weights of the original CLIP model as the teacher model.

[0084] The present invention also provides a graphic processing system based on a lightweight CLIP model, including a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0085] Many components in an electronic device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.

[0086] The computing unit can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as image processing methods and / or model training methods. For example, in some embodiments, the image processing methods and / or model training methods can be implemented as computer software programs that are tangibly contained in machine-readable media, such as storage units. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of the image processing methods and / or model training methods described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to perform image processing methods and / or model training methods in any other appropriate manner (e.g., by means of firmware).

[0087] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0089] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0090] Example 2

[0091] In order to verify the superiority of the training data set provided by the above embodiment, in this embodiment, training is performed on the traditional CC12M data set and the CC12M-SYN of the present invention respectively, and the results are as follows: Figure 4 As shown, the vertical axis refers to the total loss in training, and the horizontal axis refers to the number of training iterations. It can be seen from the figure that the model can achieve smaller loss with the same number of iterations on the dataset proposed in the present invention, that is, the dataset proposed in the present invention has higher training efficiency.

[0092] To verify the feasibility and superiority of the method provided in the above embodiment, in this embodiment, the CLIP model improved by various methods was compared with the lightweight CLIP model provided by the present invention in the context of image-text retrieval and image-text classification on various datasets. The results are shown in Tables 1 and 2. It can be seen that the method provided by the present invention is efficiently trained on a small number of datasets, thereby improving model performance.

[0093] Table 1 Comparative experimental data table under image and text retrieval

[0094]

[0095]

[0096] Table 2 Comparative experimental data table for image and text classification

[0097]

[0098]

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A graphic and text processing method based on a lightweight CLIP model, characterized in that: The method includes: Acquire image and text data to be processed, perform feature extraction on the image and text data using a lightweight CLIP model, and perform image and text retrieval or image and text classification based on the extracted features; The lightweight CLIP model is an improved CLIP model obtained by replacing the Pre-LN block in the original CLIP model with a SAS-P block containing an attention mechanism. The training of the improved model includes: The improved CLIP model is used as a student model, and the original CLIP model is used as a teacher model for model training. The model training uses contrast loss, pairing loss and multi-stage knowledge distillation loss to optimize the improved CLIP model parameters; and the multi-stage knowledge distillation includes parallel single-modal feature space distillation, contrast correlation matrix space distillation and cross-contrast space distillation.

2. The image and text processing method based on the lightweight CLIP model according to claim 1, characterized in that: The feature processing process of the SAS-P block includes: Normalizing the processed image and text features to obtain first image and text features; Processing the first image-text feature using an attention mechanism and a feedforward neural network respectively to obtain a second image-text feature and a third image-text feature; Processing the first graphic feature and the second graphic feature element by element to obtain a fourth graphic feature; The third image-text feature and the fourth image-text feature are processed element by element to obtain the output feature of the SAS-P block.

3. The image and text processing method based on the lightweight CLIP model according to claim 1, characterized in that: The training method includes: Construct a data augmentation dataset, and randomly extract a batch of images and texts from the data augmentation dataset; Input the pictures and texts into the original CLIP model to obtain teacher visual features and teacher text features, and input the same batch of pictures and texts into the improved CLIP model to obtain student visual features and student text features; Performing unimodal feature space distillation based on the teacher visual features, teacher text features, student visual features, and student text features, and calculating the unimodal feature space distillation loss; Calculating a teacher image-text correlation matrix and a student image-text correlation matrix based on the teacher visual features, the teacher text features, the student visual features, and the student text features, performing contrastive correlation matrix space distillation based on the teacher image-text correlation matrix and the student image-text correlation matrix, and calculating the contrastive correlation matrix space distillation loss; Calculating a first bimodal feature cross-correlation matrix and a second bimodal feature cross-correlation matrix based on the teacher visual features, the teacher text features, the student visual features, and the student text features, performing cross-contrast spatial distillation based on the first bimodal feature cross-correlation matrix and the second bimodal feature cross-correlation matrix, and calculating the cross-contrast spatial distillation loss; Calculate the multi-stage knowledge distillation loss based on the single-modal feature space distillation loss, contrastive correlation matrix space distillation loss, and cross-contrast space distillation loss; The multi-stage knowledge distillation loss is combined with the contrast loss and pairing loss to improve the CLIP model parameter optimization.

4. The image and text processing method based on the lightweight CLIP model according to claim 3 is characterized in that: The method for constructing a data augmentation dataset is as follows: using the CC12M dataset as an initial dataset, and using an image-text alignment model to generate multiple synthetic captions for each image in the CC12M dataset to obtain a data augmentation dataset.

5. The image and text processing method based on the lightweight CLIP model according to claim 3 is characterized in that: The calculation expression of the single-modal feature space distillation loss is: Where |B| represents the batch size; represents the teacher’s visual features of the k-th image and text; represents the student visual features of the k-th image and text; represents the teacher text features of the k-th image; Represents the student text features of the k-th image and text.

6. The image and text processing method based on the lightweight CLIP model according to claim 3 is characterized in that: The calculation expression of the contrast correlation matrix space distillation loss is: Among them, |B| represents the batch size; Sim T Represents the teacher's picture and text correlation matrix; Sim S represents the student image-text correlation matrix; KL(·) represents the KL divergence calculation.

7. The image and text processing method based on the lightweight CLIP model according to claim 3 is characterized in that: Calculate the first bimodal feature cross-correlation matrix, which is expressed as: Where |B| represents the batch size; represents the student visual features of the k-th image and text; represents the teacher text feature of the kth image; τ represents the temperature parameter; represents the teacher text features of the bth image; Calculate the second bimodal feature cross-correlation matrix, which is expressed as: in, Represents the student text features of the k-th image; represents the teacher’s visual features of the k-th image and text; Represents the teacher's visual features of the b-th image and text.

8. The image and text processing method based on the lightweight CLIP model according to claim 3 is characterized in that: The cross-comparison spatial distillation loss calculation method is: in, represents the first bimodal feature cross-correlation matrix; represents the second bimodal feature cross-correlation matrix.

9. The image and text processing method based on the lightweight CLIP model according to claim 3, characterized in that: The calculation expression of the multi-stage knowledge distillation loss function is: L KD =λ1L FD +λ2L IC +λ3L CRD , Among them, λ1, λ2, and λ3 are all hyperparameters; L FD represents the single-modal feature space distillation loss; L IC represents the cross-contrast spatial distillation loss; L CRD represents the cross-contrast spatial distillation loss.

10. A graphic and text processing system based on a lightweight CLIP model, characterized in that: The system is used to implement the method according to any one of claims 1 to 9.

Images