First view video description system based on retrieval enhancement

Abstract

The invention belongs to the technical field of video processing, and particularly relates to a first view angle video description system based on retrieval enhancement. The system comprises a cross-view retrieval module, a video encoder and a cross-modal text decoder, and the cross-view-angle retrieval module aligns visual and text features of the first view angle and the third view angle through the constructed video pairing data of the first view angle and the third view angle, so that the cross-modal and cross-view-angle retrieval of the third view angle video data of the first view angle is realized to assist the generation of the first view angle video description. The video encoder performs space-time cooperative feature extraction on the video, and the feature compression sampler performs dimension reduction on the video features; and the cross-modal text decoder adopts a large language model to be combined with a cross-modal attention layer, continuously focuses visual modal information on an input text to realize cross-modal information interaction, and outputs a first visual angle video description. According to the method, the video and text information of the third view angle is utilized, and only a small amount of third view angle data is used as assistance, so that the first view angle video description accuracy can be remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of video processing, and in particular relates to a first-person perspective video description system based on retrieval enhancement. Background Art

[0002] In recent years, deep learning has achieved gratifying results in a variety of computer vision applications. As one of the most worthy problems to explore, video description generation is an important research direction in the intersection of computer vision and natural language processing. Its goal is to automatically generate content-related natural language descriptions from a given video clip. Research in this field involves multiple key issues, including video feature extraction, semantic understanding, action recognition, sequence modeling, and natural language generation. At the same time, first-person video understanding has attracted increasing attention, such as in action recognition, detection, video text retrieval, localization, and eye movement estimation. Since first-person videos are usually recorded from a "first-person" perspective and reflect activities that interact with the environment, they play a vital role in deploying visual models to real-world scenarios such as robotics and augmented reality. As the initial baseline solution for first-person video understanding, early studies directly trained models on first-person videos. However, these methods are limited by the size of the dataset and ignore the benefits of leveraging the large corpus of third-person videos. To exploit the information in third-person videos, some researchers have tried to directly transfer models learned from third-person videos to first-person videos. However, due to differences in recording angles, camera motion, video continuity, and domain shift, video representations learned from third-person videos may not be the most appropriate for first-person videos. Another area of ​​research involves joint learning of first-person and third-person video representations. However, training these models requires time-synchronized first- and third-person video pairs collected in the same environment, which dramatically increases the cost of data acquisition. Because humans have the ability to learn by observation, that is, by observing demonstrations from a third-person perspective and then generalizing to the first-person perspective in different environments. More specifically, people often search for solutions to problems before attempting unfamiliar tasks and watch third-person instructional videos when attempting tasks, such as how to repair a bicycle chain. When it comes to performing tasks, people also recall key steps, visual demonstrations, and detailed instructions from instructional videos. Summary of the invention

[0003] The purpose of the present invention is to provide a first-perspective video description system based on retrieval enhancement, so as to realize the knowledge transfer from the third-perspective video to enhance the video understanding of the first-perspective.

[0004] The first-person perspective video description system based on retrieval enhancement provided by the present invention first introduces an automated data generation pipeline to construct paired first-person perspective and third-person perspective paired data sets. The core method is to semantically align subtitles describing similar actions. Subsequently, these paired first-person perspective and third-person perspective videos are used to train a cross-view retrieval module through the cross-view EgoExoNCE loss. This loss function aligns the first-person perspective and third-person perspective video features by sharing text features that describe similar action semantics, thereby aligning the two perspectives through text. With the help of this cross-view retrieval capability, a first-person perspective video subtitle model based on retrieval enhancement is designed using the retrieved third-person perspective video as a reference.

[0005] In the present invention, the automated data generation pipeline extracts the description of each video in the first and third perspective video datasets, and extracts the verbs and nouns in the descriptions using dependency analysis. The first perspective video and the third perspective video with the same verbs and nouns become paired videos. Based on this strategy, the first perspective video dataset and the third perspective video dataset that are independently and asynchronously collected can be paired.

[0006] The system of the present invention specifically includes a cross-view retrieval module, a video encoder, and a cross-modal text decoder. The cross-view retrieval module includes a cross-view video feature extractor and a text encoder. Through the constructed first-view and third-view video pairing data, the visual and text features of the first view and the third view are aligned, and the third view video data is retrieved cross-modally and cross-view according to the first view to assist in the generation of the first-view video description; the video encoder includes a pre-trained Transformer visual encoder and a feature compression sampler, which uses self-attention to extract spatiotemporal collaborative features from the video, and reduces the dimension of the video features through the feature compression sampler; the cross-modal text decoder uses a large language model combined with a cross-modal attention layer, so that the input text continuously focuses on the visual modality information to achieve cross-modal information interaction, and the large language model is decoded to output the first-view video description. The present invention uses the video and text information of the third-view video to assist the generation of the first-view video text description with the ability of cross-view sample retrieval, and can significantly improve the accuracy of the first-view video description under the premise of using only a very small amount of third-view data as an auxiliary.

[0007] (i) The cross-view retrieval module is composed of a first-view visual feature extractor, a third-view visual feature extractor, a cross-view visual feature extractor, and a text encoder. The cross-view retrieval module uses paired first-view and third-view data to complete training through a cross-view information alignment objective function.

[0008] First, the first-view visual feature extractor and text encoder extract the first-view video features and text features respectively, which are expressed as:

[0009]

[0010] In the formula, Respectively represent the original first-person video, the text description corresponding to the original first-person video, the first-person video representation, and the text representation corresponding to the first-person video; f ego ,f text They are the first-person visual feature extractor and the text encoder respectively; T and d represent the number of video frames and the dimension of the feature respectively. The first-person visual feature extractor is composed of a visual Transformer model trained on first-person video-text pair data. In this stage, the parameters of the first-person visual feature extractor are frozen and do not change.

[0011] Similarly, using the third-view visual feature extractor f exo , using the f used above text As a text encoder, the third-view video features and text features are extracted, expressed as:

[0012]

[0013] In the formula, They represent the original third-person video, the text description corresponding to the original third-person video, the third-person video representation, and the text representation corresponding to the third-person video. The third-person visual feature extractor is composed of a visual Transformer model trained on the third-person video-text pair data.

[0014] Based on the extracted first-view and third-view video representations, a cross-view video feature extractor is used to map the first-view video features and the third-view video features into a unified visual representation space independent of the view. The mapping process is expressed as follows:

[0015]

[0016] Where φ(·) represents the cross-view video feature extractor.

[0017] Based on visual and textual representations, the present invention defines the following three positive sample pairs for cross-view information alignment:

[0018] (1) First-person video text pair And the third-person video text pair——

[0019] (2) Pairing first-person video with paired third-person text and third-person video paired with paired first-person text The construction method is that the paired third-person perspective videos and first-person perspective videos have related verbs and nouns, and thus describe similar actions and can be used as similar sample pairs;

[0020] (3) A video-text pair from any perspective if and only if they contain the same nouns and verbs.

[0021] Based on the above three criteria, the positive sample set is defined as follows:

[0022]

[0023] Where N(t) and V(t) represent the noun and verb sets describing t, respectively; B represents the number of samples in a training batch.

[0024] Using the constructed positive sample pair set, the mapped first-perspective video and third-perspective video representation are represented using the cross-perspective information alignment objective function. Combined with the corresponding text representation The cross-view retrieval module is trained to bring all samples in the positive sample pair set closer together, while the distances of the remaining pair combinations in a training batch are pulled further apart. Specifically defined as:

[0025]

[0026] In the formula,<z,u> Represents the cosine similarity of two features. After training, the cross-view retrieval module can be used to retrieve semantically related third-view videos and texts based on the first-view video. Specifically, given a first-view video, by calculating the average similarity between the first-view video and all candidate third-view videos and texts, the K with the highest similarity are taken as the retrieval results. The retrieval process can be expressed as:

[0027]

[0028] Next, the present invention realizes cross-modal and cross-perspective retrieval of third-perspective video data based on the first-perspective based on the retrieved semantically related third-perspective video and text to assist in generating the first-perspective video description.

[0029] In fact, the video encoder (including the feature compression sampler) and the cross-modal text decoder constitute a video description model, which enables the input text to aggregate visual modality information to achieve cross-modal information interaction, and decodes and outputs the first-person video description.

[0030] (ii) The video encoder includes a pre-trained Transformer visual encoder and a feature compression sampler; by taking the first-view and the retrieved related third-view video as input, after video segmentation and position tag encoding, their features are extracted respectively. The encoding process can be expressed as:

[0031] z=Φ v (v)∈R (K+1)×T×N×d #(11)

[0032] In the formula, represents the concatenation result of K retrieved third-perspective videos and first-perspective videos; z represents the spatiotemporal representation of all videos; and N represents the number of video blocks into which each video is cut.

[0033] The feature compression sampler uses the output features of the encoder as its input and aggregates and compresses the features to achieve dimensionality reduction. Specifically, the feature compression sampler reduces the dimensions of T×N visual features to L. The feature compression sampler uses the Transformer decoder structure and introduces L learnable visual representations through the cross-attention mechanism, which continuously interact and aggregate with the T×N visual features to achieve feature dimensionality reduction. The process can be expressed as:

[0034]

[0035] Among them, pe represents temporal position encoding.

[0036] (III) A cross-modal text decoder obtains the compressed first-view and third-view video features. It decodes and outputs the text description of the first-view video based on the text of the input third-view video. The decoder is composed of a large language model combined with a cross-modal cross-attention layer. It uses text information as a query and visual information as a key-value pair, and aggregates visual information from text. Among them, each text word only focuses on the features of the corresponding video, and does not explicitly interact with other video features. Define the text input as:

[0037]

[0038] in, <video>and <eoc>They represent special video representations respectively; specifically, in the cross-modal text decoder, a gated cross-attention module is inserted between every two original language model blocks. Compared with the standard language model block, the gated cross-attention module embodies two key changes: (1) The cross-attention layer uses text features as queries and visual features as keys and values ​​to achieve visual conditional generation. Each text tag only cross-attentions the visual features of its corresponding video. (2) A Tanh gating mechanism is added. The output of each cross-attention layer and feedforward network in the cross-attention block is multiplied by a Tanh layer initialized with zero, thereby ensuring that the decoder is initially equivalent to the original language model and improving training stability. The video description generation model is trained by optimizing the standard cross-entropy loss for each predicted tag.

[0039] The workflow of the retrieval-enhanced first-person video description system of the present invention can be summarized as follows:

[0040] (I) First, through an automated process, paired first-person and third-person video datasets are constructed based on the co-occurrence relationship of similar nouns and verbs. Specifically, the massive third-person video data available on the Internet is screened and paired with the first-person video data. The description corresponding to each video is extracted to construct positive sample pairs. The text of the third-person video data is rewritten in a standardized manner through a language model to match the text description format in the first-person video.

[0041] (ii) Based on paired first-perspective and third-perspective video data, a cross-perspective retrieval module is trained.

[0042] (iii) For any given first-perspective data, a cross-perspective retrieval module is used to retrieve semantically related third-perspective video data, and a retrieval-enhanced video description model is trained.

[0043] Advantages of the present invention include:

[0044] (1) Design an automated data processing pipeline that does not require any manually labeled paired first-person and third-person video data, and instead uses an automated pipeline to construct paired data on independent first-person and third-person video datasets.

[0045] (2) For the first time, a cross-view retrieval model is proposed, multi-granularity positive samples are designed to construct positive sample pairs, and a cross-view alignment objective function is proposed to train the cross-view retrieval model.

[0046] (3) For the first time, a Transformer-based retrieval-enhanced first-person video description model, called EgoInstructor, is proposed. It can make full use of the retrieved third-person video information as an aid to generate more accurate first-person video descriptions. Its core is to automatically retrieve semantically strongly related third-person videos and text data based on a given first-person video through a cross-modal video retrieval module. This module implements cross-modal and cross-person video retrieval through an objective function that aligns cross-person information.

[0047] (4) The present invention achieves leading video retrieval and video description results on public datasets such as EPIC-Kitchen, Ego4d, CharadesEgo, YouCook2 and EgoLearner. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a block diagram of the first-person perspective video description system based on retrieval enhancement of the present invention.

[0049] Figure 2 It is a structural diagram of the first-person perspective video description system based on retrieval enhancement of the present invention. DETAILED DESCRIPTION

[0050] As can be seen from the background technology, the problems faced by most previous studies are: most of the existing first-person video understanding work focuses on using only first-person video data, ignoring the massive third-person video text data on the Internet; at the same time, there is a lack of a large number of paired cross-person data sets, which makes it impossible for the model to learn cross-person video representation alignment on a large scale. The present invention conducts in-depth research on the above problems and provides a first-person video description system based on retrieval enhancement to automatically construct paired first-person and third-person data and train a cross-person retrieval module. Given the first-person data, it can automatically retrieve related third-person video text data and output a more accurate first-person video description.

[0051] The present invention is further described below by way of examples in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, the first-view video description system based on retrieval enhancement of the present invention includes a cross-view retrieval module, a video encoder, and a cross-modal text decoder. The cross-view retrieval module includes a cross-view video feature extractor and a text encoder, specifically:

[0053] 1: The cross-view retrieval module is composed of a first-view visual feature extractor, a third-view visual feature extractor, a cross-view visual feature extractor, and a text encoder. Specifically, the first-view visual feature extractor, the third-view visual feature extractor, and the text encoder extract the first-view video, the third-view video, the text of the first-view video, and the text representation of the third-view video respectively as follows:

[0054]

[0055] The model outputs the encoded video features and text representation. Next, for the first-view video features and the third-view video features, a cross-view video feature extractor is used to map the first-view video features and the third-view video features to a unified visual representation space that is independent of the view, expressed as:

[0056]

[0057] The model outputs a visual representation that is independent of viewpoint and

[0058] Second: Based on visual and textual representations, the present invention defines the following three positive sample pairs for cross-perspective information alignment: (1) first-perspective video-text pairs, and third-perspective video-text pairs; (2) first-perspective video and paired third-perspective text pairs, and third-perspective video and paired first-perspective text pairs; (3) video-text pairs from any perspective, if and only if they contain the same nouns and verbs. Based on the above three criteria, the positive sample set is defined as follows:

[0059]

[0060] Using the constructed positive sample pair set, the mapped first-view video and third-view video representations are trained with the cross-view information alignment objective function, which is specifically defined as:

[0061]

[0062] After the cross-view module training is completed, the most similar samples are selected according to the cross-view video-text similarity and video-video similarity. The retrieval process can be expressed as:

[0063]

[0064] 3. Based on the selected third-perspective videos and corresponding texts, a mixed video input of the first and third perspectives is constructed, and the visual encoder extracts dense visual features from the first and third perspective videos respectively. The text input prompt is the corresponding description of all third-perspective videos:

[0065] z=Φ v (v)∈R (K+1)×T×N×d ,#(11)

[0066]

[0067] Fourth: Based on dense visual features, a feature compression sampler is used to aggregate and compress features to achieve dimensionality reduction. Specifically, the feature compression sampler reduces the dimensions of T×N visual features to L. The feature compression sampler uses the Transformer decoder structure and introduces L learnable visual representations through the cross-attention mechanism, which continuously interact and aggregate with the T×N visual features to achieve feature dimensionality reduction. The process can be expressed as:

[0068]

[0069] Finally, a cross-modal text decoder is used to obtain the compressed first-view and third-view video features. It decodes and outputs the first-view video text description based on the text of the third-view video input.

[0070] In summary, the present invention aims at the problem of first-person video description, attempts to utilize the semantic information of massive third-person videos and text descriptions on the Internet, and proposes a novel retrieval-enhanced first-person video description model, called EgoInstructor. It can automatically retrieve semantically related third-person videos and use them to assist in generating first-person video descriptions. The model includes a cross-person retrieval module, a video encoder, and a cross-modal text decoder, making it possible to use large-scale third-person video data on the Internet to assist in understanding the content of first-person videos.

[0071] First, the present invention obtains first-person videos from the Ego4d dataset and third-person videos from the HowTo100M dataset, and constructs first-third-person video pairs offline for training the cross-person retrieval module. Secondly, the videos in the Ego4d cooking subset, as well as paired third-person videos and subtitles, are used to train a retrieval-enhanced subtitle model. In the inference stage, given a first-person video, the present invention retrieves related third-person videos through the cross-person retrieval module and uses them as references to generate subtitles for the first-person video.

[0072] In the retrieval module, the first-view video encoder uses the VideoMAE-L model pre-trained on Ego4d. The third-view video encoder uses the UniformerV2 model and uses the InternVideo weights pre-trained on the third-view video dataset. Both video encoders output a 768-dimensional feature vector, and the input is 8 frames of video. The present invention decodes the video at a rate of 8 frames per second, obtaining one feature per second. To match the average duration of Ego4d and HowTo100M, 4 frame features are randomly sampled by default and input into the cross-view video encoder. For the constructed long video-text pairs, long video clips between 60 and 300 seconds are cropped, including 20 narrations. The large language model is tuned using the Stable-Vicuna-v1.5 instruction for long text summarization and subtitle optimization. The cross-view video encoder consists of 4 layers of Transformer encoder layers. The text encoder is a BERT model. The retrieval module is trained for 5 epochs, using the AdamW optimizer, a learning rate of 3×10^-5, and a batch size of 4096. The batch size setting in the retrieval module should be dynamically adjusted according to the hardware GPU memory. This project was completed using Nvidia A10080G GPU. The batch size can be appropriately reduced to 1024 or 512 on GPUs with smaller memory.

[0073] For the retrieval-enhanced caption model, the CLIP-L / 14 visual encoder is used. The Perceiver resampler consists of 6 layers of Transformer blocks with 64 learnable query features. Two language model variants, LLaMA-7B and MPT-1B, are used. To achieve fast convergence, the caption model is initialized with pre-trained weights. The model is trained for 3 epochs with an initial learning rate of 1×10^-5 and a cosine learning rate decay. The batch size for the LLaMA and MPT models is set to 8 and 32, respectively.

[0074] Cross-view retrieval experimental results:

[0075] Subtitle optimization effect: Through the comparison in Table 2 (ID-3 vs. ID-4, ID-6 vs. ID-7), it can be seen that the optimized third-perspective video subtitles have consistent improvements in multiple evaluation indicators compared to the original speech recognition transcription. This shows that the optimized subtitles are better aligned with the visual content. There is also a 1% to 2% performance improvement on the first-perspective benchmark test set EgoMCQ. This is because through subtitle optimization, the first-perspective and third-perspective video subtitle formats are better aligned, allowing the first-perspective video to be matched with more semantically rich subtitles, thereby enhancing the expression of the first-perspective video-text relationship. In the cross-perspective benchmark EgoLearner video retrieval, the performance has been consistently improved under the Ego2Exo setting, because the semantically similar first-perspective and third-perspective videos are better aligned through optimized subtitles.

[0076] EgoExoNCE loss function effect: In order to study the impact of different loss functions in aligning first-person and third-person videos, EgoExoNCE is compared with two commonly used loss functions, InfoNCE and EgoNCE. As shown in Table 2, EgoNCE improves on the first-person benchmark (ID-3 vs. ID-5) compared to InfoNCE, that is, on the EK100-MIR and EgoMCQ tasks, but only slightly improves on the cross-person benchmark. In contrast, EgoExoNCE improves the accuracy by an average of 2.0% and 4.3% on the cross-person benchmark (ID-3 vs. ID-6), namely the EgoLearner and CharadesEgo tasks. The results can be further improved by using improved subtitles (ID-4 vs. ID-7). This shows that the EgoExoNCE loss function is better at mining cross-person samples using subtitles, thereby enhancing the model's video alignment capabilities. In summary, our cross-view visual alignment method outperforms the baseline model and variants of various related methods (ID-3 vs. ID-7) on all benchmark tests.

[0077] Analysis of first-perspective video encoders: The present invention uses a variety of pre-trained encoders to extract first-perspective video features, including CLIP, EgoVLP, LaViLa, InternVideo, and VideoMAE, while ensuring the consistency of the third-perspective video features extracted by InternVideo. The results are shown in Table 3. All baseline models are trained on the original first-perspective and third-perspective data using the InfoNCE loss function. By adding the cross-perspective visual representation alignment strategy of the present invention, that is, training with optimized paired data and EgoExoNCE loss, all models show consistent improvements in most performance indicators. Specifically, on the cross-perspective evaluation benchmark, both CLIP and InternVideo models have good improvements. For example, the model of the present invention improves the InternVideo baseline model by 15.9% and 7.7% on the CharadesEgo task, respectively.

[0078] Table 2. Ablation experiments, text descriptions of third-view instructional videos (original ASR transcription vs. optimized subtitles) and cross-view alignment loss functions (InfoNCE vs. EgoNCE vs. EgoExoNCE).

[0079]

[0080] Table 3. Impact of cross-view visual representation alignment strategies on different types of first-view video encoders.

[0081]

[0082] Experimental results on retrieval-enhanced video description

[0083] Impact of external third-person videos: Table 4 shows the impact of introducing retrieval samples. The number of external third-person videos K used was varied from 0 to 8, also known as K-shot evaluation. In the 0-shot scenario, i.e., without using any external third-person videos, both LLaMA- and MPT-based video description models achieved low performance, with the MPT-based model only achieving a CIDER score of 0.468. In contrast, using only 1 external third-person video as a reference, the model showed significant performance improvements on all metrics. As the number of retrieval samples increased, the description performance also gradually improved, indicating that the multimodal description model can successfully utilize the relevant visual and textual information of the external third-person video. Compared with the modified description, using the original speech recognition text as the input prompt will cause the LLaMA-based description model to perform slightly worse (CIDER 1.665 lower than 1.768). A previous first-person video caption model LaViLA-GPT2 was also re-implemented, and the retrieval-enhanced description model of the present invention significantly surpassed LaViLA in performance.

[0084] Table 4. Video captioning results on the Ego4d validation set (cooking subset). Shot refers to the number of external third-person perspective videos used.

[0085]

[0086] Experimental results on the EgoLearner dataset: The generalization ability of the EgoInstructor model trained on the Ego4d cooking subset is evaluated by directly migrating it to the EgoLearner cooking videos. As shown in Table 5, in the 0-shot scenario, the description model of the present invention cannot achieve satisfactory results, which may be due to the differences in the video and subtitle distributions of the two datasets. In the 1-shot scenario, a significant performance improvement can be observed, which shows that EgoInstructor can effectively transfer knowledge from the retrieved external third-person videos to help the subtitle generation of the first-person videos, thereby achieving a fast adaptation from Ego4d to EgoLearner. In addition, two variants of retrieval samples are studied: (1) randomly selected and (2) external third-person videos paired with the ground truth. Table 5 shows that the performance of using randomly selected videos is similar to that of 0-shot subtitles, indicating that random samples cannot provide useful semantic information and the model may be misled by irrelevant reference videos. In contrast, the samples obtained by the cross-view retriever of the present invention can bring a significant improvement in description performance, and the performance is better when using external third-person videos paired with the ground truth.

[0087] Table 5. Zero-shot and one-shot video description results on EgoLearner. All models are directly transferred to EgoLearner after being trained on the Ego4d cooking subset.

[0088] < / eoc> < / video>

Claims

1. A first-person video description system based on retrieval enhancement, characterized in that: Firstly, an automated data generation pipeline is introduced to construct paired first-person and third-person perspective datasets; Using these paired first- and third-view videos, the cross-view retrieval module is trained using the cross-view EgoExoNCE loss. This loss function aligns the first- and third-view video features by sharing text features that describe similar action semantics, thereby aligning the two views through text. With the help of cross-view retrieval capability, we use the retrieved third-person video as a reference to design a retrieval-enhanced first-person video captioning model. The automated data generation pipeline extracts the description of each video in the first- and third-person video datasets, and uses dependency analysis to extract the verbs and nouns in the descriptions; first-person videos and third-person videos with the same verbs and nouns become paired videos. The first-perspective video description system specifically includes a cross-perspective retrieval module, a video encoder, and a cross-modal text decoder; the cross-perspective retrieval module includes a cross-perspective video feature extractor and a text encoder, which aligns the visual and text features of the first and third perspectives through the constructed first-perspective and third-perspective video pairing data, and realizes cross-modal and cross-perspective retrieval of third-perspective video data based on the first perspective to assist in generating first-perspective video description; the video encoder includes a pre-trained Transformer visual encoder and a feature compression sampler, which uses self-attention to realize spatiotemporal collaborative feature extraction of the video, and reduces the dimension of the video features through the feature compression sampler; the cross-modal text decoder adopts a large language model combined with a cross-modal attention layer, so that the input text continuously focuses on the visual modality information to realize cross-modal information interaction, and the large language model is used to decode and output the first-perspective video description.

2. The first-person video description system according to claim 1, characterized in that: The cross-view retrieval module is composed of a first-view visual feature extractor, a third-view visual feature extractor, a cross-view visual feature extractor, and a text encoder; the cross-view retrieval module uses paired first-view and third-view data to complete training through a cross-view information alignment objective function; First, the first-view visual feature extractor and text encoder extract the first-view video features and text features respectively, which are expressed as: In the formula, Respectively represent the original first-person video, the text description corresponding to the original first-person video, the first-person video representation, and the text representation corresponding to the first-person video; f ego ,f text are the first-person visual feature extractor and text encoder respectively; T and d represent the number of video frames and the dimension of features respectively; Similarly, using the third-view visual feature extractor f exo , using the f used above text As a text encoder, the third-view video features and text features are extracted, expressed as: In the formula, Respectively represent the original third-person perspective video, the text description corresponding to the original third-person perspective video, the third-person perspective video representation, and the text representation corresponding to the third-person perspective video; Based on the extracted first-view and third-view video representations, a cross-view video feature extractor is used to map the first-view video features and the third-view video features into a unified visual representation space independent of the view. The mapping process is expressed as follows: Where φ(·) represents the cross-view video feature extractor; Based on visual and textual representations, the following three positive sample pairs are defined for cross-view information alignment: (1) First-person video text pair And the third-person video text pair—— (2) Pairing first-person video with paired third-person text and third-person video paired with paired first-person text The construction method is that the paired third-person perspective videos and first-person perspective videos have related verbs and nouns, and thus describe similar actions and can be used as similar sample pairs; (3) A video-text pair from any perspective is a pair if and only if they contain the same nouns and verbs; Based on the above three criteria, the positive sample set is defined as follows: Where N(t) and V(t) represent the noun and verb sets describing t, respectively; B represents the number of samples in a training batch; Using the constructed positive sample pair set, the mapped first-view video and third-view video representation are represented using the cross-view information alignment objective function. Combined with the corresponding text representation The cross-view retrieval module is trained to bring all samples in the positive sample pair set closer together, while the distances of the remaining pair combinations in a training batch are pulled further apart. Specifically defined as: In the formula,<z,u> Represents the cosine similarity of two features; after training, the cross-view retrieval module retrieves semantically related third-view videos and texts based on the first-view video; specifically, given a first-view video, by calculating the average similarity between the first-view video and all candidate third-view videos and texts, the K with the highest similarity are taken as the retrieval results; the retrieval process is expressed as: Based on the retrieved semantically related third-perspective videos and texts, cross-modal and cross-perspective retrieval of third-perspective video data based on the first-perspective is achieved to assist in generating first-perspective video descriptions.

3. The first-person perspective video description system according to claim 2, characterized in that: The video encoder includes a pre-trained Transformer visual encoder and a feature compression sampler; by obtaining the first-view and the retrieved related third-view videos as input, after video segmentation and position mark encoding, the features are extracted respectively; The encoding process is expressed as: z=Φ v (v)∈R (K+1)×T×N×d #(11) In the formula, represents the splicing result of K retrieved third-perspective videos and first-perspective videos; z represents the spatiotemporal representation of all videos; N represents the number of video blocks each video is cut into; The feature compression sampler takes the output features of the visual encoder as its input, aggregates and compresses the features to achieve dimensionality reduction; specifically, the feature compression sampler reduces the dimensions of T×N visual features to L; the feature compression sampler adopts the Transformer decoder structure, introduces L learnable visual representations through the cross-attention mechanism, and continuously interacts and aggregates with the T×N visual features to achieve feature dimensionality reduction; the process is expressed as: Among them, pe represents temporal position encoding.

4. The first-person perspective video description system according to claim 3, characterized in that: The cross-modal text decoder, after obtaining the compressed first-view and third-view video features, decodes and outputs the first-view video text description based on the text of the third-view video as an aid; the decoder is composed of a large language model combined with a cross-modal cross-attention layer, taking text information as a query and visual information as a key-value pair, and aggregating visual information from text; wherein each text word only focuses on the features of the corresponding video without explicitly interacting with other video features; the text input is defined as: in, <video>and <eoc> They represent special video representations respectively.< / eoc> < / video> 5. The first-person perspective video description system according to any one of claims 1 to 4, characterized in that: In the cross-view retrieval module, the first-view visual feature extractor is composed of a visual Transformer model trained on first-view video-text pair data; the third-view visual feature extractor is composed of a visual Transformer model trained on third-view video-text pair data.

6. The first-person perspective video description system according to any one of claims 1 to 4, characterized in that: In the cross-modal text decoder, a gated cross-attention module is inserted between every two original language model blocks; wherein the cross-attention layer uses text features as queries and visual features as keys and values ​​to achieve visual conditional generation; each text tag only performs cross-attention with the visual features of its corresponding video; a Tanh gating mechanism is added, and the output of each cross-attention layer and feedforward network in the cross-attention block is multiplied by a Tanh layer initialized with zero, thereby ensuring that the decoder is initially equivalent to the original language model and improving training stability; the video description generation model is trained by optimizing the standard cross-entropy loss of each predicted tag.

7. The first-person perspective video description system according to claim 6, characterized in that: The workflow is: (i) First, through an automated process, paired first-person and third-person video datasets are constructed based on the co-occurrence relationship of similar nouns and verbs; specifically, the massive third-person video data available on the Internet is screened and paired with the first-person video data, and positive sample pairs are constructed by extracting the description corresponding to each video. The text of the third-person video data is rewritten in a standardized manner through a language model to make it consistent with the text description format in the first-person video; (2) training a cross-view retrieval module based on paired first-view and third-view video data; (iii) For any given first-perspective data, a cross-perspective retrieval module is used to retrieve semantically related third-perspective video data, and a retrieval-enhanced video description model is trained.

Images