A multi-granularity knowledge distillation-based cross-modal video retrieval method and system

Abstract

The application relates to a multi-granularity knowledge distillation-based cross-modal video retrieval method and system, characterized in that the method comprises the following steps: determining a text to be queried; inputting the text to be queried into a student model of a pre-input video to output corresponding videos; the student model is obtained by training a teacher model by using a multi-granularity teaching training algorithm; while the lower calculation and storage overhead of the student model is reserved, the retrieval precision gap between the student model and the teacher model is narrowed, and the application can be widely applied to the field of cross-modal video retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cross-modal video retrieval, and particularly relates to a cross-modal video retrieval method and system based on multi-granularity knowledge distillation. BACKGROUND

[0002] Cross-modal video retrieval aims to enable general users to perform content-based retrieval on a large number of unlabeled videos by using text queries in natural language form given on the spot.

[0003] The prior art discloses a CLIP4Clip model, which first extracts image features frame by frame for a video using the visual encoder of the CLIP model, then enhances the image features using a plurality of layers of Transformer modules, and then fuses the enhanced frame-level features into a video-level feature using average pooling. The above-mentioned video-level feature can be extracted and stored in advance, so that the CLIP4Clip model only needs to compare the query feature with the video feature in the database during the retrieval stage, and therefore the model has high efficiency for the cross-modal video retrieval task. To further improve the retrieval performance, the prior art also discloses X-CLIP, TS2-Net and X-Pool, which consider more fine-grained interactive content, such as calculating the similarity between frame-level features and text features.

[0004] Although the retrieval accuracy of these models is relatively higher, the storage and computing overhead required by the fine-grained interaction of these models is much larger than that of the CLIP4Clip model, which restricts the application of these models in large-scale cross-modal video retrieval tasks, and it is difficult to balance accuracy and efficiency. SUMMARY

[0005] To solve the above problems, the present application aims to provide a cross-modal video retrieval method and system based on multi-granularity knowledge distillation, which can balance accuracy and efficiency.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: in a first aspect, a cross-modal video retrieval method based on multi-granularity knowledge distillation is provided, comprising:

[0007] determining a text to be queried;

[0008] inputting the text to be queried into a student model pre-inputted with a video, and outputting a plurality of videos corresponding thereto;

[0009] The student model is obtained by training a teacher model using a multi-granularity teaching algorithm.

[0010] Further, the construction process of the student model is:

[0011] Based on the CLIP4Clip model, a student model is constructed;

[0012] Based on the CLIP text video retrieval model, a teacher model is constructed;

[0013] Obtain video text pair data as a training set, select the parameters of the teacher model, and set the number of iterations;

[0014] Input the training set into the constructed student model and teacher model, and the teacher model automatically generates a supervision signal to guide the training of the student model. The student model is iteratively optimized through back propagation to obtain a trained student model;

[0015] Batch input videos in the video library into the pre-constructed student model, and each video generates a fixed-length video-level feature.

[0016] Further, the student model comprises:

[0017] A visual encoder for generating corresponding frame-level features {v1,..., v m} from m video frames {f1,..., f m}, each frame-level feature being a d-dimensional real number vector;

[0018] A converter module for enhancing the frame-level features {v1,..., v m} to generate enhanced frame-level features {φ1,..., φ m};

[0019] A frame-level feature fusion module based on an attention mechanism for receiving a fine-grained teaching loss from the teacher model during the student model training stage, generating an m-dimensional non-negative weight vector {w1,..., w m}, where w i represents the importance of video frame f i , i = 1, 2,..., m, and the non-negative weight vector {w1,..., w m} is used to weight and fuse the enhanced frame-level features {φ1,..., φ m} to generate a d-dimensional video-level feature φ(x);

[0020] A text encoder for generating a d-dimensional text feature vector φ(t) from the input text t;

[0021] A cross-modal similarity calculation module for calculating the cosine similarity between the text feature vector φ(t) and the video-level feature φ(x).

[0022] Further, the training set is input into the constructed student model and teacher model, the teacher model automatically generates a supervision signal for guiding the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model, comprising:

[0023] The training set is used to train the constructed teacher model to obtain a trained teacher model.

[0024] The multi-granularity teaching training algorithm is used, the teacher model automatically generates two supervision signals of fine-grained teaching loss and coarse-grained teaching loss for guiding the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model.

[0025] Further, the multi-granularity teaching training algorithm is used, the teacher model automatically generates two supervision signals of fine-grained teaching loss and coarse-grained teaching loss for guiding the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model, comprising:

[0026] Given a set of video-text pair data, the teacher model gives video-level relevance scores and frame-level relevance scores about the text, and the student model gives cosine similarity between the text feature vector and the video-level feature and a non-negative weight vector;

[0027] The coarse-grained teaching loss between the video-level relevance score given by the teacher model and the cosine similarity given by the student model is calculated;

[0028] The fine-grained teaching loss of the frame-level relevance score given by the teacher model and the non-negative weight vector given by the student model is calculated;

[0029] Based on the cosine similarity given by the student model, a symmetric InfoNCE contrastive learning loss is calculated.

[0030] The parameters of the student model are iteratively optimized according to the obtained coarse-grained teaching loss, fine-grained teaching loss and InfoNCE contrastive learning loss using the back propagation algorithm.

[0031] Further, the text to be queried is input into the student model of the pre-input video, and a corresponding number of videos are output, comprising:

[0032] The text to be queried is input into the text encoder of the student model of the pre-input video to generate a fixed-length text feature vector;

[0033] The cross-modal similarity calculation module of the student model calculates the cosine similarity between the text feature vector and the pre-extracted video-level feature as a measure of the semantic relevance between the text to be queried and the video;

[0034] According to the calculated cosine similarity, output a number of most relevant videos in the video library.

[0035] In a second aspect, a cross-modal video retrieval system based on multi-granularity knowledge distillation is provided, comprising:

[0036] A data acquisition module is configured to determine a text to be queried.

[0037] A video retrieval module is configured to input the text to be queried into a student model of a pre-input video, and output a number of corresponding videos, wherein the student model is obtained by training a teacher model using a multi-granularity teaching algorithm.

[0038] Further, the system further comprises a model construction module, wherein the model construction module comprises:

[0039] A student model construction unit is configured to construct a student model based on a CLIP4Clip model.

[0040] A teacher model construction unit is configured to construct a teacher model based on a CLIP text-video retrieval model.

[0041] A training set determination unit is configured to obtain video-text pair data as a training set, select parameters of the teacher model, and set an iteration number.

[0042] A student model training unit is configured to input the training set into the constructed student model and teacher model, and the teacher model automatically generates a supervision signal to guide the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model.

[0043] A video input unit is configured to batch input videos in a video library into the pre-constructed student model, and each video generates a fixed-length video-level feature.

[0044] In a third aspect, a processing device is provided, comprising computer program instructions, wherein the computer program instructions are used to implement corresponding steps of the cross-modal video retrieval method based on multi-granularity knowledge distillation when the processing device executes the computer program instructions.

[0045] In a fourth aspect, a computer readable storage medium is provided, wherein the computer readable storage medium stores computer program instructions, and the computer program instructions are used to implement corresponding steps of the cross-modal video retrieval method based on multi-granularity knowledge distillation when the processor executes the computer program instructions.

[0046] The present application has the following advantages due to the above technical solutions:

[0047] 1. The application provides a knowledge distillation scheme based on multi-granularity teaching, which narrows the retrieval accuracy gap between the student model and the teacher model while retaining the lower computational and storage overhead of the student model.

[0048] 2. Unlike the CLIP4Clip model using average pooling to de-fuse video frame features, the student model of the application is provided with a frame-level feature fusion module based on an attention mechanism, which considers the differences in the importance of different video frames and can achieve better video retrieval accuracy.

[0049] 3. The frame-level feature fusion module based on the attention mechanism provided in the student model of the application can solve the problem of the student model lacking the ability to accept multi-granularity knowledge from the teacher model.

[0050] 4. The application only requires the teacher model to provide video-level and frame-level relevance scores for the query text, and does not make specific requirements for the internal implementation of the teacher model. Therefore, the application can use a single teacher model or a combination of multiple teacher models to guide the student model, and has a wide range of applications.

[0051] 5. The application uses a pre-trained teacher model, and the multi-granularity relevance scores of the teacher model for a specific video and a specific query text can be calculated offline before training the student model. Therefore, in terms of training the student model, the training time and memory overhead of multi-granularity knowledge distillation are comparable to standard training strategies.

[0052] In summary, the application can be widely used in the field of cross-modal video retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0053] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments, and are not meant to limit the present application. Throughout the drawings, the same reference designates the same elements. In the drawings:

[0054] Figure 1 is the architecture schematic diagram of the student model and the teacher model in the method provided by an embodiment of the application. DETAILED DESCRIPTION

[0055] Exemplary embodiments of the present application will be described hereinafter with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be accurately conveyed to those skilled in the art.

[0056] It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order

[0057] Although the terms first, second, third, and the like can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can be only used to differentiate one element, component, region, layer or section from another region, layer or section. Terms such as "first", "second", and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

[0058] To maintain the retrieval efficiency comparable to the CLIP4Clip model, while trying to narrow the retrieval accuracy gap between the CLIP4Clip model and the latest fine-grained interaction model as much as possible, the embodiment of the present application proposes a multi-granularity knowledge distillation based cross-modal video retrieval method and system. The present application distills the video-level coarse-grained knowledge and frame-level fine-grained knowledge in the larger and stronger teacher model into the student model based on CLIP4Clip in the form of soft video-text relevance labels and frame-level weights, respectively, so as to achieve a balance between retrieval efficiency and retrieval accuracy. Prior to the present application, the TeachText model proposed in the prior art has applied the knowledge distillation method to the cross-modal video retrieval task. The TeachText model first trains a series of text-video retrieval models using different text encoders, and then uses the average value of the video-text similarity given by these models as a soft label to supervise the student model training. The present application has the following two significant differences from the TeachText model in the technical solution: 1) The TeachText model only considers video-level coarse-grained knowledge, and its solution cannot be used for frame-level fine-grained knowledge distillation, while the present application first proposes to use frame-text relevance as a supervision signal to guide the attention module in the student model to give appropriate frame-level weights. 2) The TeachText model uses pre-extracted expert features, and its retrieval performance is significantly inferior to that of the CLIP-based cross-modal video retrieval model, while the present application is based on the CLIP4Clip model, and its end-to-end deep learning architecture has significantly better learning ability than the TeachText. To enable the student model to simultaneously accept video-level and frame-level cross-modal multi-granularity knowledge from the teacher model, the present application introduces an attention mechanism based frame-level feature fusion module based on the CLIP4Clip model. The frame-level weight calculation of this module can be calculated offline, so that the calculation and storage overhead of the student model in the retrieval stage is still comparable to that of the original CLIP4Clip model, while the retrieval accuracy is significantly better than that of the CLIP4Clip model. The present application solves the problem of how to distill fine-grained knowledge from the teacher model to the student model, and solves the problem of how to balance accuracy and efficiency in the existing CLIP-based cross-modal video retrieval model.

[0059] Embodiment 1

[0060] As Figure 1 shown, the present embodiment provides a multi-granularity knowledge distillation based cross-modal video retrieval method, which comprises the following steps:

[0061] 1) Pre-constructing a student model and a teacher model, specifically:

[0062] 1.1) Constructing a student model based on a CLIP4Clip model.

[0063] Specifically, the student model includes a visual encoder, a transformer module, an attentional feature fusion (AFA) module, a text encoder, and a cross-modal similarity calculation module.

[0064] A visual encoder is used to encode m video frames {f1, ..., f2}. m} Generate the corresponding frame-level features {v1, ..., v m Each frame-level feature is a d-dimensional real vector, where m video frames {f1, ..., f2} represent the features. m The frame is extracted from video x (uniformly sampled along the time axis) using standard video frame extraction tools.

[0065] The converter module is used to process frame-level features {v1, ..., v} m} is enhanced to produce enhanced frame-level features {φ1, ..., φ m}, where the number of converter modules is four.

[0066] The attention-based frame-level feature fusion module is used during the student model training phase to receive fine-grained teaching loss from the teacher model and generate an m-dimensional non-negative weight vector {w1, ..., w...}. m}, where w i Indicates video frame f i The importance of i = 1, 2, ..., m, the non-negative weight vector {w1, ..., w m} is used for the enhanced frame-level features {φ1, ..., φ m The weighted fusion is performed to produce a d-dimensional video-level feature φ(x), i.e. Specifically, fine-grained knowledge is used as supervisory information to constrain the generation of non-negative weight vectors.

[0067] A text encoder is used to generate a d-dimensional text feature vector φ(t) based on the input text t.

[0068] The cross-modal similarity calculation module is used to calculate the cosine similarity between the text feature vector φ(t) and the video-level feature φ(x) to measure the semantic relevance between the text query and the video. This calculated cosine similarity is then passed as a parameter, along with the cosine similarity generated by the teacher model, to the coarse-grained teaching loss calculation module of the teacher model for calculating the coarse-grained teaching loss.

[0069] Specifically, the frame-level feature fusion module based on the attention mechanism includes a first linear layer, a ReLU activation layer, a second linear layer, and a Softmax layer.

[0070] The first linear layer is used for the preliminary linear mapping of the features; the ReLU activation layer is used to remove the negative values in the first linear layer, retain the positive values unchanged, improve the sparsity of the model, and enhance the nonlinear ability of the model; the second linear layer is used to map the d-dimensional video-level features into l-dimensional video-level features, and the Softmax layer is used to map the l-dimensional video-level features into a non-negative weight vector in the interval (0, 1).

[0071] Specifically, the frame-level weight calculation of the frame-level feature fusion module based on the attention mechanism can be performed offline, so that the calculation and storage overhead of the student model is still comparable to that of the original CLIP4Clip model in the retrieval stage, and the retrieval accuracy of the AFA module is significantly better than that of the CLIP4Clip model due to the consideration of the difference in importance of different video frames.

[0072] Specifically, the data flow of the student model is as follows:

[0073]

[0074] 1.2) CLIP-based text video retrieval model, construct a teacher model.

[0075] Specifically, although the CLIP-based text video retrieval model (for example: X-CLIP, TS2-Net, X-Pool) improves the retrieval accuracy by introducing more fine-grained interactions, the fine-grained interactions also bring huge storage and calculation overhead, which limits the application of these models in large-scale video retrieval tasks. Therefore, the present application adopts a multi-granularity knowledge distillation scheme, using these models with higher retrieval accuracy but higher overhead as teacher models to teach student models. The multi-granularity knowledge distillation is embodied in two forms. One is to distill the video-level coarse-grained teaching loss in the teacher model to the student model in the form of soft video-text relevance labels (i.e. Figure 1 Coarse-grained teaching in the Fine-grained teaching in the Figure 1 Compared with the coarse-grained teaching of TeachText, the multi-granularity teaching training algorithm used in the present application can further reduce the performance gap between the student model and the teacher model.

[0076] Specifically, in this embodiment, the structure of the teacher model is not fixed and is flexible and diverse, which can be determined according to specific conditions, which is one of the advantages of the present application.

[0077] 1.3) Obtain video-text pair data, select the parameters of the teacher model, and set the iteration number and the parameters of the optimizer, etc.

[0078] 1.4) Take the obtained several video-text pair data (v, t) as the training set, where v is the video data and t is the text data.

[0079] 1.5) Input the training set into the constructed student model and teacher model using the multi-granularity teaching training algorithm, the teacher model automatically generates a supervision signal to guide the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model:

[0080] 1.5.1) Train the constructed teacher model using the training set to obtain a trained teacher model.

[0081] 1.5.2) Use the multi-granularity teaching training algorithm, the teacher model automatically generates two supervision signals of fine-grained teaching loss and coarse-grained teaching loss to guide the training of the student model, and the student model is iteratively optimized through back propagation to obtain a trained student model:

[0082] 1.5.2.1) Given a set of video-text pair data {(v, t)}, the teacher model gives the video-level relevance score and frame-level relevance score about the text, and the student model gives the cosine similarity between the text feature vector and the video-level feature (i.e. the video-level relevance score about the text) and an m-dimensional non-negative weight vector.

[0083] 1.5.2.2) Calculate the Pearson distance loss between the video-level relevance score given by the teacher model and the video-level relevance score given by the student model, i.e. the coarse-grained teaching loss.

[0084] 1.5.2.3) Calculate the cross-entropy loss between the frame-level relevance score given by the teacher model and the non-negative weight vector given by the student model, i.e. the fine-grained teaching loss.

[0085] 1.5.2.4) Based on the video-level relevance score given by the student model, calculate the symmetric InfoNCE contrastive learning loss.

[0086] 1.5.2.5) Use the back propagation algorithm to iteratively optimize the parameters of the student model according to the obtained cross-entropy loss, Pearson loss and InfoNCE contrastive learning loss.

[0087] Specifically, only the lightweight student model needs to be retained after training, and the teacher model with high computational and storage overhead does not need to be retained.

[0088] 2) Pre-input the videos in the video library into the pre-constructed student model in batches, and each video generates a fixed-length video-level feature.

[0089] 3) input the text to be queried into the student model of the pre-input video, and output several most relevant videos, specifically:

[0090] 3.1) input the text to be queried into the text encoder of the student model of the pre-input video, to generate a fixed-length text feature vector.

[0091] 3.2) the cross-modal similarity calculation module of the student model calculates the cosine similarity according to the text feature vector and the pre-extracted video-level feature, as a measure of the semantic relevance of the text to be queried and the video.

[0092] 3.3) according to the calculated cosine similarity, output several most relevant videos in the video library.

[0093] The effectiveness of the method of the present application is described in detail below through specific embodiments:

[0094] First, an ablation experiment is performed on a widely used cross-modal video retrieval dataset MSRVTT, and the MSRVTT dataset has two division methods MSRVTT-1k and MSRVTT-3k. Since the MSRVTT-1k division is more widely used, the ablation experiment of the present application is performed on MSRVTT-1k, and then comparative experiments with other cross-modal video retrieval methods are performed on more extensive datasets, including MSRVTT, MSVD, VATEX and ActivityNet-Caption (ActNetCap).

[0095] For the ablation experiment on MSRVTT-1k, the present application uses the most commonly used top-k recall Rk (k = 1, 5, 10) and the SumR index that can evaluate the overall performance, and the larger the Rk and SumR values, the better the model retrieval performance. For comparative experiments with other cross-modal video retrieval methods, the present application reports the FLOPs index representing the model calculation overhead and the Store index representing the storage overhead in addition to the SumR index of each model on all datasets. The smaller the values of the FLOPs index and the Store index, the better.

[0096] The ablation experiment data is shown in Table 1 as follows, wherein #0 represents the teacher model X-CLIP, #1 and #2 represent the performance of the student model without using knowledge distillation, #1 is the CLIP4Clip model, #2 is the student model of the application, that is, the AFA module is added on the basis of the CLIP4Clip model, #3, #4 and #5 represent the performance of the student model when the X-CLIP model is used as the teacher model and the student model designed by the application is used for knowledge distillation. Among them, #3 is to use only coarse-grained knowledge distillation, #4 is to use only fine-grained knowledge distillation, and #5 is the multi-grained knowledge distillation of the application. Compared with #3 and #4, the performance of #5 is improved, which shows that the multi-grained knowledge distillation scheme proposed by the application can further reduce the gap between the student model and the teacher model. Compared with #1 and #2, the AFA module proposed by the application not only enables the student model to accept fine-grained knowledge from the teacher network, but also improves the retrieval performance of the student model itself:

[0097] Table 1: Ablation experiment of multi-grained knowledge distillation and AFA module of the application on MSRVTT-1k

[0098]

[0099]

[0100] The experimental results of the comparison experiment with other cross-modal video retrieval methods are shown in Table 2. The experimental results on multiple data sets show that the application achieves a good balance between computational overhead, storage overhead and retrieval accuracy:

[0101] Table 2: Comparison experiment of the application and other cross-modal video retrieval methods on multiple data sets

[0102]

[0103] Embodiment 2

[0104] The embodiment provides a cross-modal video retrieval system based on multi-grained knowledge distillation, comprising:

[0105] The data acquisition module is configured to determine the text to be queried.

[0106] The video retrieval module is configured to input the text to be queried into the student model of the pre-input video, and output a plurality of corresponding videos, wherein the student model is trained by the teacher model using a multi-grained teaching algorithm.

[0107] In a preferred embodiment, the model construction module further comprises:

[0108] The student model construction unit is configured to construct a student model based on a CLIP4Clip model.

[0109] The teacher model construction unit is configured to construct a teacher model based on a CLIP-based text-video retrieval model.

[0110] The training set determination unit is configured to obtain video-text pair data as a training set, select parameters of the teacher model, and set an iteration number.

[0111] The student model training unit is configured to input the training set into the constructed student model and teacher model, and automatically generate a supervision signal by the teacher model to guide the training of the student model, and iteratively optimize the student model through back propagation to obtain a trained student model.

[0112] The video input unit is configured to batch input videos in a video library into the pre-constructed student model, and generate a fixed-length video-level feature for each video.

[0113] The system provided in the embodiment is used to execute the above-mentioned method embodiments, and the specific process and detailed content are referred to the above-mentioned embodiments, which will not be described here.

[0114] Embodiment 3

[0115] The processing device provided in the embodiment 1 corresponds to the cross-modal video retrieval method based on multi-granularity knowledge distillation, and can be applied to a processing device of a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of the embodiment 1.

[0116] The processing device includes a processor, a memory, a communication interface and a bus, the processor, the memory and the communication interface are connected through the bus to complete the communication between each other. The memory stores a computer program that can run on the processing device, and the processing device runs the computer program to execute the cross-modal video retrieval method based on multi-granularity knowledge distillation provided in the embodiment 1.

[0117] In some implementations, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0118] In other implementations, the processor can be a central processing unit (CPU), a digital signal processor (DSP) or various types of general-purpose processors, which are not limited here.

[0119] In addition, the logic instructions in the memory described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0120] Those skilled in the art can understand that the structure of the computing device described above is only part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. A specific computing device can include more or fewer components, or combine certain components, or have a different arrangement of components.

[0121] Embodiment 4

[0122] The embodiment provides a computer program product corresponding to the cross-modal video retrieval method based on multi-granularity knowledge distillation provided in the embodiment 1. The computer program product can include a computer readable storage medium, and the computer readable storage medium has loaded computer readable program instructions for executing the cross-modal video retrieval method based on multi-granularity knowledge distillation.

[0123] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0124] The computer readable storage medium provided in the above embodiment has similar implementation principles and technical effects to the above method embodiments, and will not be described here.

[0125] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0126] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0127] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 one or more flow or blocks.

[0128] The above-described embodiments are merely intended to illustrate the present application, and the structure, connection manner, and manufacturing process of each component can be changed. Any equivalent changes and improvements made on the basis of the technical solutions of the present application should not be excluded from the protection scope of the present application.

Claims

1. A cross-modal video retrieval method based on multi-granularity knowledge distillation, characterized in that, include: Determine the text to be queried; Input the text to be queried into the student model that has been pre-entered with videos, and output the corresponding videos; The student model is obtained by training the teacher model using a multi-granularity teaching training algorithm; The process of constructing the student model is as follows: A student model is constructed based on the CLIP4Clip model; A teacher model is constructed based on the CLIP-based text-video retrieval model. Obtain video text pairs as the training set, select the parameters of the teacher model, and set the number of iterations; The training set is input into the constructed student model and teacher model. The teacher model automatically generates supervision signals to guide the training of the student model. The student model is iteratively optimized through backpropagation to obtain a well-trained student model. The videos in the video library are input in batches into the pre-built student model, and each video generates a fixed-length video-level feature; The process involves inputting the training set into the constructed student model and teacher model. The teacher model automatically generates supervision signals to guide the training of the student model. The student model undergoes iterative optimization through backpropagation to obtain a trained student model, including: The constructed teacher model is trained using the training set to obtain a well-trained teacher model; A multi-granularity teaching and training algorithm is adopted. The teacher model automatically generates two kinds of supervision signals, fine-grained teaching loss and coarse-grained teaching loss, to guide the training of the student model. The student model is iteratively optimized through backpropagation to obtain a well-trained student model. The algorithm employs a multi-granularity teaching and training approach. The teacher model automatically generates two types of supervisory signals: fine-grained teaching loss and coarse-grained teaching loss, to guide the training of the student model. The student model undergoes iterative optimization through backpropagation to obtain a well-trained student model, including: Given a set of video-text pairs, the teacher model provides video-level relevance scores and frame-level relevance scores for the text, while the student model provides cosine similarity and non-negative weight vectors between the text feature vector and the video-level features. Calculate the coarse-grained teaching loss between the video-level relevance score given by the teacher model and the cosine similarity given by the student model; Calculate the fine-grained teaching loss between the frame-level relevance score given by the teacher model and the non-negative weight vector given by the student model; Based on the cosine similarity given by the student model, calculate the symmetric InfoNCE contrastive learning loss; The backpropagation algorithm is used to iteratively optimize the parameters of the student model based on the obtained coarse-grained teaching loss, fine-grained teaching loss, and InfoNCE contrastive learning loss.

2. The cross-modal video retrieval method based on multi-granularity knowledge distillation as described in claim 1, characterized in that, The student model includes: A visual encoder, used to... video frames Generate corresponding frame-level features Each frame-level feature is a A dimensional real vector; The converter module is used to adjust frame-level features. Enhancement is performed to generate enhanced frame-level features. ; A frame-level feature fusion module based on an attention mechanism is used during the student model training phase to receive fine-grained teaching loss from the teacher model and generate a... dimensional nonnegative weight vector ,in, Represents video frames The importance of The non-negative weight vector Used for enhanced frame-level features Perform weighted fusion to produce a Dimensional video-level features ; A text encoder, used to decode input text. , produce a 3D text feature vector ; The cross-modal similarity calculation module is used to calculate text feature vectors. With video-level features Cosine similarity between them.

3. The cross-modal video retrieval method based on multi-granularity knowledge distillation as described in claim 2, characterized in that, The process of inputting the text to be queried into a student model that has pre-input videos and outputting several corresponding videos includes: The text to be queried is input into the text encoder of the student model that has been pre-input video, which produces a fixed-length text feature vector. The cross-modal similarity calculation module of the student model calculates cosine similarity based on text feature vectors and pre-extracted video-level features, which is used to measure the semantic relevance between the text and the video to be queried. Based on the calculated cosine similarity, output the most relevant videos in the video library.

4. A cross-modal video retrieval system based on multi-granularity knowledge distillation, based on the cross-modal video retrieval method based on multi-granularity knowledge distillation as described in any one of claims 1 to 3, characterized in that, include: The data acquisition module is used to determine the text to be queried; The video retrieval module is used to input the text to be queried into a student model that has been pre-input videos and output several corresponding videos. The student model is trained by the teacher model using a multi-granularity teaching training algorithm.

5. The cross-modal video retrieval system based on multi-granularity knowledge distillation as described in claim 4, characterized in that, It also includes a model building module, which includes: The student model building unit is used to build student models based on the CLIP4Clip model. The teacher model building unit is used to build teacher models based on the CLIP-based text-video retrieval model. The training set determination unit is used to acquire video text pair data as the training set, select the parameters of the teacher model, and set the number of iterations; The student model training unit is used to input the training set into the constructed student model and teacher model. The teacher model automatically generates supervision signals to guide the training of the student model. The student model is iteratively optimized through backpropagation to obtain a well-trained student model. The video input unit is used to input videos from the video library in batches into the pre-built student model, and each video generates a fixed-length video-level feature.

6. A processing apparatus, characterized in that, It includes computer program instructions, wherein when executed by a processing device, the computer program instructions are used to implement the steps corresponding to the cross-modal video retrieval method based on multi-granularity knowledge distillation as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, wherein when executed by a processor, the computer program instructions are used to implement the steps corresponding to the cross-modal video retrieval method based on multi-granularity knowledge distillation as described in any one of claims 1-3.

Images