Improved visual encoder, visual language model and video text retrieval method

Abstract

The invention provides an improved visual encoder, a visual language model and a video text retrieval method. The improved visual encoder comprises a ViT architecture, and an event activation module is arranged between a multi-head attention layer and a normalization layer of a top ViT block of the ViT architecture; the event activation module comprises an initialization module, an alignment module and a perfection module, the multi-head attention layer of the top ViT block outputs a self-attention mapping graph, a class lexical element sequence and an image block lexical element sequence of each video to the initialization module, and the initialization module outputs an event activation graph of each video to the alignment module; and the alignment module outputs the aligned event activation graph of each video to the perfection module, and the perfection module outputs the activated image block lexical elements of each video to the normalization layer of the top ViT block. According to the method, the dynamic evolution process of the event in the video can be effectively captured, and the retrieval accuracy and robustness are effectively improved.

Description

Technical Field

[0001] This invention relates to an improved visual encoder, visual language model, and video text retrieval method, belonging to the field of information retrieval technology. Background Technology

[0002] As more and more videos are uploaded to the Internet, video text retrieval has quickly come into the researchers' view. Its purpose is to find the most suitable results in a database containing a large number of videos based on a text query.

[0003] In video text retrieval, existing technologies generally employ large-scale pre-trained visual language models (such as the CLIP model) as their infrastructure. A typical implementation process for this type of technology is as follows: First, the Visual Transformer (ViT) model is used to encode each frame of the video sequence, extracting frame-level feature representations for each frame. Then, specific pooling operations are used to aggregate these frame-level features into a global video-level feature representation, which is then used for similarity matching with the features of the text query to obtain retrieval results. However, this approach ignores the temporal dependencies between video frames, resulting in an inability to effectively capture the dynamic evolution of events in the video, thus affecting retrieval accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide an improved visual encoder, visual language model, and video text retrieval method to solve the problems existing in the prior art.

[0005] According to a first aspect of the present invention, an improved visual encoder is provided, comprising: a ViT architecture, wherein an event activation module is set between the multi-head attention layer and the normalization layer of the top-level ViT block of the ViT architecture; the event activation module includes an initialization module, an alignment module, and a refinement module; the multi-head attention layer of the top-level ViT block outputs a self-attention map S for each video and a word class sequence X for each video. cls and the image block word sequence X for each video p In the initialization module, the self-attention map S of each video is composed of the self-attention map matrix of each frame of the corresponding video. The initialization module outputs the event activation map A of each video to the alignment module, and the image block word sequence X of each video. p The alignment module outputs the aligned event activation map A' of each video to the refinement module, which in turn outputs the activated image block terms for each video. The normalization layer up to the top-level ViT block contains T frames of images for each video.

[0006] The initialization module is configured with an algorithm that extracts the first row of the self-attention mapping matrix of each frame of each video to form the initialization event activation map A0 of the corresponding video, and reshapes the initialization event activation map A0 of each video into the event activation map A of each video.

[0007] The alignment module is equipped with an algorithm that uses the Hungarian matching algorithm to align the event activation maps of all frames in the event activation map A of each video to obtain the aligned event activation map A' of each video.

[0008] The refinement module is configured to correct the aligned event activation map A' of each video to obtain the refined first event activation map for the corresponding video. The momentum-based algorithm, with its enhancement module configured to enhance the first event activation graph of each video, is described. Reshape the event activation graph for each video into a more refined version. The event activation graph for each video is then generated by successively multiplying the elements. Image block word sequence X corresponding to the video p Coupled to obtain the activated image block lexical sequence for each video. The algorithm;

[0009] in, , N represents the number of image blocks divided into each frame, D represents the feature dimension of the word class, and R represents the set of real numbers;

[0010] The activated image block word sequence of each video after being processed sequentially by the normalization layer of the top-level ViT block and the feedforward neural network. The word sequence X corresponding to the video cls Aggregate to generate a video representation E for each video. v .

[0011] In one specific implementation, the algorithm for obtaining the aligned event activation map A' for each video is as follows: For the current video, obtain the aligned event activation map for each frame of the current video. The set of aligned event activation maps for all frames of the current video forms the aligned event activation map A' for the current video. Repeat the above algorithm to obtain the aligned event activation map A' for each video. The algorithm for obtaining the aligned event activation map for each frame of the current video is as follows: The alignment module uses the Hungarian matching algorithm to find the image block word sequence associated with the j-th frame of the current video. Image block word sequence of the (j-1)th frame of the current video Optimal matching mapping , match mapping Reshaping makes the matching map Based on matching mapping Generate the aligned event activation graph of the j-th frame of the current video. ,in, Each element can take values ​​from 1 to N. This represents the j-1th frame of the current video. Image block words, M represents the i-th image block word in the j-th frame of the current video. j This is an index value that temporarily binds 1 to N, where cos(·,·) represents cosine similarity. , .

[0012] In one specific implementation, the step of obtaining the refined event activation graph for each video... The algorithm is as follows: For the current video, obtain the improved event activation map for each frame of the current video. The set of improved event activation maps for all frames of the current video forms the improved event activation map of the current video. Repeat the above algorithm to obtain the improved event activation map A' for each video. The algorithm for obtaining the improved event activation map for each frame of the current video is as follows: Use the structured information of salient events in the aligned event activation maps of all frames preceding the current frame of the current video to correct the aligned event activation map of the current frame of the current video, thus obtaining the improved event activation map for the current frame of the current video. , In this case, ⊙ indicates that each element is multiplied digit by digit. Let γ be the learnable weight matrix, γ = GELU(f(A')), GELU(·) is the activation function, and f(·) is a function containing 2 transformers in each head; For the f-th component in γ, This is the (f-1)th component in γ, and so on. It is the first component of γ; This is the aligned event activation graph for the f-th frame of the current video. This is the aligned event activation map for the (f-1)th frame of the current video, and so on. This is the aligned event activation graph of the first frame of the current video.

[0013] In one specific implementation, the activated image block lexical sequence for each video is obtained. The algorithm is as follows: Γ represents the improved first event activation graph for each video. Remodeled into the improved first event activation image corresponding to the video. The reshaping function.

[0014] According to a second aspect of the present invention, a visual language model includes: a CLIP model architecture, wherein the visual encoder of the CLIP model architecture is replaced with an improved visual encoder as described above, while the other parts of the CLIP model architecture remain unchanged.

[0015] According to a third aspect of the present invention, a video text retrieval method is provided, which performs video text retrieval by inputting multiple videos and texts to be retrieved into a visual language model trained as described above.

[0016] In one specific implementation, during the training process of the visual language model, the AdamW optimizer is used to adjust the weights to be trained in the visual language model, with a weight decay coefficient set to 0.2. A cosine decay strategy is used for learning rate scheduling, and the learning rate of the CLIP model architecture's initialization parameters is set to 1×10⁻⁶. -6 The learning rate of the event activation module's initialization parameters is set to 1×10. -4 The loss function uses symmetric cross-entropy loss.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] 1. The improved visual encoder proposed in this invention captures the temporal dependencies of salient event regions by inserting an event activation module into the top-level ViT block of the ViT architecture, without requiring additional event localization models and annotations. The visual language model proposed in this invention uses the improved visual encoder as its visual encoder. Extensive experiments were conducted on the MSR-VTT dataset, DiDeMo dataset, and LSMDC dataset to verify the performance of the improved visual encoder and the visual language model proposed in this invention. The verification results show that both the improved visual encoder and the visual language model proposed in this invention have excellent effectiveness and generalization, and their performance is significantly improved.

[0019] 2. Video text retrieval using the visual language model proposed in this invention is effective because the visual language model can effectively capture the dynamic evolution of events in the video, thus improving its retrieval accuracy and robustness. In particular, it still has excellent retrieval accuracy and robustness when retrieving videos containing complex dynamic events. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the improved visual encoder proposed in this invention.

[0021] Figure 2 This is a model architecture diagram of the visual language model proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the embodiments and accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0023] refer to Figure 1 According to a first aspect of the present invention, an improved visual encoder includes: a ViT architecture, wherein an event activation module is disposed between the multi-head attention layer and the normalization layer of the top-level ViT block of the ViT architecture, the event activation module being used to capture the temporal dependencies of salient event regions; in this embodiment, the top-level ViT architecture is layer L; the event activation module includes an initialization module, an alignment module, and a refinement module; the multi-head attention layer of the top-level ViT block outputs a self-attention map S for each video and a word class sequence X for the video set. cls and the image block word sequence X of the video set p In the initialization module, the self-attention map S of each video is composed of the self-attention map matrix of each frame of the corresponding video. The initialization module outputs the event activation map A of each video to the alignment module, and the image block word sequence X of each video. p The alignment module outputs the aligned event activation map A' of each video to the refinement module, which in turn outputs the activated image block terms for each video. The normalization layer up to the top-level ViT block contains T frames of images for each video.

[0024] The improved visual encoder sequentially performs the initialization, alignment, and refinement of the event activation map for each video, thereby modeling the temporal dependencies of video events and significantly improving the discriminative ability of video representation.

[0025] The initialization module is configured with an algorithm that extracts the first row of the self-attention mapping matrix of each frame of each video to form the initial event activation map A0 of each video, and reshapes the initial event activation map A0 of each video into the event activation map A of each video.

[0026] By leveraging the self-attention mechanism within ViT, a preliminary event activation map for each video is automatically generated from the attention relationship between class tokens and patch tokens in each video, which provides the beneficial effect of directly and efficiently locating salient event regions within the frame.

[0027] The alignment module is equipped with an algorithm that uses the Hungarian matching algorithm to align the event activation maps of all frames in the event activation map A of each video to obtain the aligned event activation map A' of each video.

[0028] The alignment module solves the problem of inconsistent spatial positions of the same event area caused by object movement between adjacent frames, ensuring the continuity of event clues in the temporal dimension, and laying an accurate foundation for the subsequent generation of a complete event activation map for each video and accurate encoding of video temporal dependencies.

[0029] Specifically, the algorithm for obtaining the aligned event activation map A' for each video is as follows: For the current video, obtain the aligned event activation map for each frame of the current video. The set of aligned event activation maps for all frames of the current video forms the aligned event activation map A' for the current video. Repeat the above algorithm to obtain the aligned event activation map A' for each video. The algorithm for obtaining the aligned event activation map for each frame of the current video is as follows: The alignment module uses the Hungarian matching algorithm to find the image block word sequence associated with the j-th frame of the current video. Image block word sequence of the (j-1)th frame of the current video Optimal matching mapping , match mapping Reshaping makes the matching map Based on matching mapping Generate the aligned event activation graph of the j-th frame of the current video. ,in, Each element can take values ​​from 1 to N. This represents the j-1th frame of the current video. Image block words, M represents the i-th image block word in the j-th frame of the current video. j This is an index value that temporarily binds 1 to N, where cos(·,·) represents cosine similarity. , ;

[0030] The refinement module is configured to correct the aligned event activation map A' of each video to obtain a refined first event activation map for each video. The momentum-based algorithm, with its enhancement module configured to enhance the first event activation graph of each video, is described. Reshape the event activation graph for each video into a more refined version. The event activation graph for each video is then generated by successively multiplying the elements. Image block word sequence X corresponding to the video p Coupled to obtain the activated image block lexical sequence for each video. The algorithm;

[0031] Specifically, the process of obtaining the refined event activation graph for each video... The algorithm is as follows: For the current video, obtain the improved event activation map for each frame of the current video. The set of improved event activation maps for all frames of the current video forms the improved event activation map of the current video. Repeat the above algorithm to obtain the improved event activation map A' for each video. The algorithm for obtaining the improved event activation map for each frame of the current video is as follows: Use the structured information of salient events in the aligned event activation maps of all frames preceding the current frame of the current video to correct the aligned event activation map of the current frame of the current video, thus obtaining the improved event activation map for the current frame of the current video. , In this case, ⊙ indicates that each element is multiplied digit by digit. Let γ be the learnable weight matrix, γ = GELU(f(A')), GELU(·) is the activation function, and f(·) is a function containing 2 transformers in each head; For the f-th component in γ, This is the (f-1)th component in γ, and so on. It is the first component of γ; This is the aligned event activation graph for the f-th frame of the current video. This is the aligned event activation map for the (f-1)th frame of the current video, and so on. This is the aligned event activation graph of the first frame of the current video.

[0032] A momentum-based strategy algorithm is used to model the temporal evolution of each video. Global weights learned by a small converter are used to optimize and fuse the aligned event activation maps of the current and historical frames for each video. This effectively strengthens persistent salient event regions and suppresses noise, thereby generating a more robust event representation.

[0033] Specifically, obtain the activated image block lexical sequence for each video. The algorithm is as follows: Γ represents the improved first event activation graph for each video. Remodeled into the improved first event activation image corresponding to the video. The reshaping function.

[0034] in, , N represents the number of image blocks divided into each frame, D represents the feature dimension of the word class, and R represents the set of real numbers;

[0035] The activated image block word sequence of each video after being processed sequentially by the normalization layer of the top-level ViT block and the feedforward neural network. The word sequence X corresponding to the video cls Aggregate to generate a video representation E for each video. v .

[0036] According to a second aspect of the present invention, a visual language model is provided, comprising: a CLIP model architecture, wherein the visual encoder of the CLIP model architecture is replaced with an improved visual encoder as described above, while the other parts of the CLIP model architecture remain unchanged.

[0037] According to a third aspect of the present invention, a video text retrieval method is provided, which performs video text retrieval by inputting multiple videos and texts to be retrieved into the above-mentioned visual language model after training.

[0038] Specifically, during the training process of the visual language model, the AdamW optimizer is used to adjust the weights to be trained in the visual language model, with a weight decay coefficient set to 0.2. The learning rate scheduling adopts a cosine decay strategy, and the initialization parameter learning rate of the CLIP model architecture is set to 1×10. -6 The learning rate of the event activation module's initialization parameters is set to 1×10. -4 The loss function uses symmetric cross-entropy loss.

[0039] The present invention will be further described in detail and its performance verified through specific embodiments to facilitate understanding of the essence of the invention. Experimental and control groups were established based on the MSR-VTT dataset, DiDeMo dataset, and LSMDC dataset, respectively.

[0040] Experimental Group: Multiple videos and texts to be retrieved were input into the visual language model provided by this invention for video-text retrieval. This model uses X-Pool as its basic framework. X-Pool is an advanced video-text retrieval framework based on the CLIP pre-trained model and employing a text-video cross-attention mechanism for feature aggregation. This experimental group replaced the original visual encoder with the improved visual encoder provided by this invention to verify the improved visual encoder's ability to model temporal dependencies in videos and its effect on improving retrieval performance.

[0041] Control group 1: Multiple videos and texts to be retrieved were input into the Frozen model for video-text retrieval.

[0042] Control group 2: Input multiple videos and texts to be retrieved into the CLIP4Clip-meanP model or the CLIP4Clip model for video and text retrieval.

[0043] Control group 3: Multiple videos and texts to be retrieved were input into the X-Pool model for video-text retrieval.

[0044] For each video to be retrieved, the following data preprocessing was performed: First, 12 frames of images were obtained by uniform sampling; then, each frame of image was scaled to a short side of 256 pixels and cropped in the center to obtain an image with a resolution of 224×224 pixels. All frame images of each video to be retrieved were used as the standardized input of the visual encoders of the experimental group and the three control groups.

[0045] In the experimental group, the improved visual encoder had an image patch size of 32×32 and a generated feature embedding dimension of 512. Multiple preprocessed videos and texts were input into the trained visual language model provided by this invention for video-text retrieval. For each input video, the event activation module of the improved visual encoder performed the following processing: 1. The multi-head attention layer of the top-level ViT block outputs the self-attention map S for each video and the word class sequence X for each video. cls and the image block word sequence X for each video p The initialization module generates an event activation map A for each video and outputs it to the alignment module. The event activation map A for each video captures the long-distance dependencies between its class lexical sequence and the corresponding image patch lexical sequence; 2. The image patch lexical sequence X for each video. p The alignment module outputs the data to the refinement module. The alignment module generates an aligned event activation map A' for each video and outputs it to the refinement module. The refinement module generates a refined event activation map for each video. The event activation graph for each video is then generated by successively multiplying the elements. Image block word sequence corresponding to the video Coupled to obtain the activated image block lexical sequence for each video. And the activated image block word sequence for each video. The output is fed to the normalization layer of the top-level ViT block. This is the sequence of activated image block terms for each video. The video continues processing after passing through the normalization layer and feedforward neural network in the current ViT block. The improved visual encoder then processes the word class sequence for each video. The corresponding activated image block word sequence of the video Aggregate to obtain the video representation E for each video. v .

[0046] The visual language model provided by this invention receives text queries and outputs text embedding representations. A similarity calculator calculates the video representation E for each video. v The cosine similarity between the video and the embedded text representation is used to obtain the retrieval score, thus completing the video text retrieval.

[0047] Table 1. Performance Metrics Comparison of Models Based on MSR-VTT Dataset

[0048] Table 2. Performance Metrics Comparison of Models Based on the DiDeMo Dataset

[0049] Table 3. Performance Metrics Comparison of Models Based on LSMDC Dataset

[0050] In Tables 1 to 3, * indicates a reimplemented method; R@K (Recall at K) represents the recall rate among the top K results; MdR (Median Rank) represents the median rank; SumR is the sum of R@1, R@5, and R@10, used to comprehensively evaluate model performance. A higher SumR and a lower MdR indicate better retrieval performance. The data in Tables 1 to 3 verify the effectiveness and advancement of the visual language model provided by this invention in video text retrieval tasks.

[0051] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions and substitutions can be made without departing from the inventive concept, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An improved visual encoder, characterized in that, include: The ViT architecture sets up an event activation module between the multi-head attention layer and the normalization layer of the top-level ViT block. The event activation module includes an initialization module, an alignment module, and a refinement module. The multi-head attention layer of the top-level ViT block outputs the self-attention map S and the word class sequence X for each video. cls and the image block word sequence X for each video p In the initialization module, the self-attention map S of each video is composed of the self-attention map matrix of each frame of the corresponding video. The initialization module outputs the event activation map A of each video to the alignment module, and the image block word sequence X of each video. p The alignment module outputs the aligned event activation map A' of each video to the refinement module, which in turn outputs the activated image block terms for each video. The normalization layer up to the top-level ViT block contains T frames of images for each video. The initialization module is configured with an algorithm that extracts the first row of the self-attention mapping matrix of each frame of each video to form the initialization event activation map A0 of the corresponding video, and reshapes the initialization event activation map A0 of each video into the event activation map A of each video. The alignment module is equipped with an algorithm that uses the Hungarian matching algorithm to align the event activation maps of all frames in the event activation map A of each video to obtain the aligned event activation map A' of each video. The refinement module is configured to correct the aligned event activation map A' of each video to obtain the refined first event activation map for the corresponding video. The momentum-based algorithm, with its enhancement module configured to enhance the first event activation graph of each video, is described. Reshape the event activation graph for each video into a more refined version. The event activation graph for each video is then generated by successively multiplying the elements. Image block word sequence X corresponding to the video p Coupled to obtain the activated image block lexical sequence for each video. The algorithm; in, , N represents the number of image blocks divided into each frame, D represents the feature dimension of the word class, and R represents the set of real numbers; The activated image block word sequence of each video after being processed sequentially by the normalization layer of the top-level ViT block and the feedforward neural network. The word sequence X corresponding to the video cls Aggregate to generate a video representation E for each video. v .

2. The improved visual encoder as described in claim 1, characterized in that, The algorithm for obtaining the aligned event activation map A' for each video is as follows: For the current video, obtain the aligned event activation map for each frame of the current video. The set of aligned event activation maps for all frames of the current video forms the aligned event activation map A' for the current video. Repeat the above algorithm to obtain the aligned event activation map A' for each video. The algorithm for obtaining the aligned event activation map for each frame of the current video is as follows: The alignment module uses the Hungarian matching algorithm to find the image block word sequence associated with the j-th frame of the current video. Image block word sequence of the (j-1)th frame of the current video Optimal matching mapping , match mapping Reshaping makes the matching map Based on matching mapping Generate the aligned event activation graph of the j-th frame of the current video. ,in, Each element can take values ​​from 1 to N. This represents the j-1th frame of the current video. Image block words, M represents the i-th image block word in the j-th frame of the current video. j This is an index value that temporarily binds 1 to N, where cos(·,·) represents cosine similarity. , .

3. The improved visual encoder as described in claim 2, characterized in that, The improved event activation graph for each video is obtained. The algorithm is as follows: For the current video, obtain the improved event activation map for each frame of the current video. The set of improved event activation maps for all frames of the current video forms the improved event activation map of the current video. Repeat the above algorithm to obtain the improved event activation map A' for each video. The algorithm for obtaining the improved event activation map for each frame of the current video is as follows: Use the structured information of salient events in the aligned event activation maps of all frames preceding the current frame of the current video to correct the aligned event activation map of the current frame of the current video, thus obtaining the improved event activation map for the current frame of the current video. , In this case, ⊙ indicates that each element is multiplied digit by digit. Let γ be the learnable weight matrix, γ = GELU(f(A')), GELU(·) is the activation function, and f(·) is a function containing 2 transformers in each head; For the f-th component in γ, This is the (f-1)th component in γ, and so on. It is the first component of γ; This is the aligned event activation graph for the f-th frame of the current video. This is the aligned event activation map for the (f-1)th frame of the current video, and so on. This is the aligned event activation graph of the first frame of the current video.

4. The improved visual encoder as described in claim 3, characterized in that, Obtain the activated image block word sequence for each video. The algorithm is as follows: Γ represents the improved first event activation graph for each video. Remodeled into the improved first event activation image corresponding to the video. The reshaping function.

5. A visual language model, characterized in that, include: The CLIP model architecture replaces the visual encoder of the CLIP model architecture with the improved visual encoder described in any one of claims 1 to 4, while the other parts of the CLIP model architecture remain unchanged.

6. A video text retrieval method, characterized in that, The visual language model of claim 5, which has been trained, is used to perform video-text retrieval by inputting multiple videos and texts to be retrieved.

7. The video text retrieval method as described in claim 6, characterized in that, During the training of the visual language model, the AdamW optimizer was used to adjust the weights to be trained in the visual language model, with a weight decay coefficient set to 0.

2. The learning rate scheduling adopted a cosine decay strategy, and the learning rate of the CLIP model architecture was set to 1×10⁻⁶ for initialization. -6 The learning rate of the event activation module's initialization parameters is set to 1×10. -4 The loss function uses symmetric cross-entropy loss.

Images